Academy

Lesson 6 — Watching It Learn

Two commands. The first trains the model and takes about three minutes. The second asks it to write.

First, open Terminal

Hold ⌘ and press Space. A search box appears in the middle of the screen.
Type Terminal and press Return. A window with plain text opens. That is Terminal.
Click Copy beside a command below, click into the Terminal window, then press ⌘V to paste.
Press Return to run it. Wait until the text stops moving before you do the next one.

1Train it

Three thousand steps. It prints the loss every 250, and saves the finished model to model.pt. Leave it alone until it stops.

cd ~/llm-from-scratch && source .venv/bin/activate && python -u gpt.py

Trains the model, then prints 500 characters it wrote.

2Ask it to write

Loads the model you just saved and lets it write on its own, one character at a time.

cd ~/llm-from-scratch && source .venv/bin/activate && python -c "
import sys, torch, gpt
m = gpt.GPT().to(gpt.device); m.load_state_dict(torch.load('model.pt', map_location=gpt.device)); m.eval()
idx = torch.zeros((1,1), dtype=torch.long, device=gpt.device)
for _ in range(420):
    logits, _u = m(idx[:, -gpt.block_size:])
    probs = torch.nn.functional.softmax(logits[:, -1, :], dim=-1)
    nxt = torch.multinomial(probs, num_samples=1)
    idx = torch.cat([idx, nxt], dim=1)
    sys.stdout.write(gpt.itos[int(nxt)]); sys.stdout.flush()
print()
"

Streams 420 characters at the speed the model actually produces them.