GPT-2 124M,
rebuilt from first principles.

GPT-2 is a decoder-only Transformer: every token can attend only to itself and earlier tokens. The implementation uses learned token and position embeddings, twelve identical pre-LN Transformer blocks, a final layer norm, and a vocabulary projection whose weights are shared with the input embedding table.

Training used the FineWeb-Edu sample-10BT configuration. Documents were encoded with the GPT-2 tiktoken vocabulary, prefixed with the end-of-text token, stored as compact uint16 arrays, and divided into 100 verified shards.

Each optimizer step aggregates 32 micro-batches of 16 sequences × 1,024 tokens, producing an effective batch of 524,288 tokens. The run uses AdamW, bfloat16 autocasting, gradient clipping, linear warmup, and cosine learning-rate decay.

Maximum learning rate

6 × 10⁻⁴

Minimum learning rate

6 × 10⁻⁵

Warmup

715 steps

Weight decay

0.1

Gradient clipping

1.0

Precision

bfloat16

Initialization seed

42

Compilation

Disabled

The final checkpoint records a training loss of 3.1033, validation loss of 3.0308, and validation perplexity of 20.71. The graphs below are generated directly from the committed CSV and clearly mark the observed continuation window.

The most consequential engineering work was not another model layer. It was making a multi-hour paid run recoverable: atomic local checkpoints, complete optimizer and RNG state, verified uploads, bounded retries, resumable downloads, and a latest pointer that advances only after integrity checks pass.

These fixed-seed samples use top-50 sampling and a 72-token total sequence limit. They are qualitative examples, not benchmark scores. Their unevenness is representative of a small, unaligned completion model.