Architecture

A decoder-only Transformer, layer by layer

The model follows the same core structure used across modern decoder-only language models, implemented directly rather than imported.

Input embeddings

Converts token IDs into dense vectors and adds positional information so the model knows token order.

Each token attends to itself and every earlier token, weighting them by relevance. The mask prevents attending to future tokens.

A position-wise fully connected network that transforms each token's representation independently.

Stabilizes training and lets gradients flow cleanly through many stacked blocks.

Maps the final hidden state back to vocabulary-sized logits for next-token prediction.