Open Source · Educational Transformer Project
A GPT-style Transformer, built from first principles.
Mini LLM From Scratch is an educational implementation of a GPT-style decoder-only Transformer built with Python and PyTorch. From tokenization and embeddings to self-attention, training, and autoregressive text generation, every core component is implemented to help learners understand how modern language models work from first principles.
Overview
Project at a glance
Everything a recruiter, student, or professor needs in under 30 seconds.
Purpose
Educational implementation of a GPT-style decoder-only Transformer
Language
Python
Framework
PyTorch
Architecture
Transformer decoder
Learning focus
Transformer internals, from first principles
Difficulty
Intermediate
License
MIT
Project type
Open source, educational project
Skills demonstrated
Motivation
Why build an LLM from scratch
01
Abstractions hide the interesting part
Calling a pretrained model teaches you how to use a Transformer. Implementing one teaches you how it actually works — tokenization, attention, and gradients included.
02
First principles transfer
Once you've written self-attention by hand, every paper, blog post, and production model built on it becomes easier to read.
03
Small enough to hold in your head
A minimal implementation removes the scale and infrastructure of production systems, leaving the core mechanics visible end to end.
Interactive
The learning journey
Step through the pipeline a token takes from raw text to generated output.
Tokenization
Raw text is split into tokens and mapped to vocabulary IDs.
Architecture
How the model is structured
Explore how each component of the decoder-only Transformer works, from embeddings to self-attention and output generation.
Explore architecturePipeline
From raw text to trained model
See how data loading, training, and checkpointing connect end to end.
Explore pipelineEngineering Overview
Engineering highlights
What each part of the repository demonstrates.
Custom Transformer implementation
The decoder-only Transformer is implemented directly in PyTorch rather than imported from a pretrained-model library, showing an understanding of each component's role.
Word-level tokenizer
A tokenizer built from scratch to convert raw text into vocabulary IDs, the first step in the model's input pipeline.
Autoregressive text generation
Implements next-token prediction and sampling, following the same fundamental autoregressive generation principle used by modern language models.
Training pipeline
A structured training loop connecting data loading, the forward pass, loss computation, and optimization into a repeatable process.
Checkpoint system
Model state can be saved and restored, a practical engineering concern for any training workflow that runs over multiple sessions.
Modular architecture
Code is organized into independent, readable components (tokenizer, embeddings, attention, blocks, training) rather than one monolithic script.
Educational documentation
Each part of the codebase is documented to explain not just what it does, but why it exists inside a Transformer.
Well-structured repository
The project follows a clean and organized structure with clear documentation, licensing, and modular code to make learning and exploration easier.
Documentation
Written like documentation, not a wiki
Every guide explains what a concept is, why it matters, and how it is implemented in this project using clear language and practical examples.
Read the documentationSupport the project
If this helped you understand how Transformer language models work
Consider giving the repository a star on GitHub.
Star on GitHub