Train an LLM from Scratch: the Full Tutorial Repo

Because the pedagogy scales down better than it scales up. Every mechanism in the pipeline, tokenization, loss masking, attention, pretraining dynamics, fine-tuning, alignment, operates identically at 13 million parameters and at 13 billion; only the hardware bill and the wall-clock change. A model too small to be useful is also too small to hide behind, meaning every design mistake shows up clearly in behavior you can inspect, while a large model’s competence can mask flawed understanding in its trainer. The small track also completes, an underrated property: a learner who finishes the whole pipeline in a weekend holds the full mental model, whereas a pipeline that requires rented clusters gets abandoned at the interesting parts. Usefulness is not the product of this exercise, comprehension is, and the small model is the right price for it.

At learning scale, the differences are visible in the training loop, which is what makes them teachable. Supervised fine-tuning alone turns a base model into something that follows chat format, competent but agreeable. Adding a reward model and DPO teaches preferences with a reference policy in the loop, which shows the learner why alignment needs something to align away from. The reference-free options, ORPO and KTO, drop the reference model, and ORPO specifically folds the fine-tuning and alignment stages into a single run, which on small hardware is the practical choice, one run instead of two, with no separate reference to store. The configuration surface makes this concrete: alignment is a loss-type flag, not a philosophy seminar. The understanding that transfers is that these algorithms are pipeline choices with cost and quality profiles, not rival ideologies, and having toggled between them on a model you built yourself is worth more than a hundred forum arguments about which is correct.

Three directions, in ascending ambition. First, scale within reach: rerun the pipeline at the largest size your hardware allows and observe what changes, which builds intuition for scaling behavior the small run cannot show. Second, change the data: swap in a domain corpus you care about and fine-tune on it, which converts the exercise from education to a small real tool, a domain-specific assistant you actually use. Third, go deeper on one stage: the repository’s steps each connect to specialist literature, and picking the stage that fascinated you most, tokenization, attention variants, alignment, is the right way to choose what to study formally next. What to avoid: mistaking completion for mastery and immediately renting a cluster to chase scale. The pipeline teaches mechanisms, and mechanisms compound; chasing parameters first mostly teaches you about cloud billing.

Almost, with one bridge to cross first. The repository names its prerequisites precisely: object-oriented programming, neural network basics, and working PyTorch, and it links starter videos for each gap. A competent programmer without the ML foundations can cross that bridge in a focused week or two and then succeed here, because the pipeline teaches the transformer as it assembles it. Someone without comfortable programming should not start here; a concepts course first, then this, is the path that ends in completion rather than frustration. The self-assessment that works: read the prerequisites list, and if each item sounds like something you could do with documentation open, you are ready; if two or more items sound like courses you have not taken, take them first, because every step of this pipeline leans on all three at once, and prerequisite gaps compound in a build, unlike in a course, where each lesson can be separately patched.

The genre has two lineages and they optimize for different lessons. The nanoGPT lineage builds a compact, elegant implementation of the core, pretraining a small GPT on a clean dataset, and its virtue is minimalism: nothing stands between you and the essential loop. This repository trades some minimalism for pipeline completeness, adding the stages a real assistant requires, chat formatting, supervised fine-tuning, reward modeling, alignment, evaluation, and a serving interface. Neither replaces the other. The minimalist build teaches the forward pass and training loop with the least distraction; the full pipeline teaches why the base model was never the product. Learners with time for both should do the minimalist build first for the foundations and this one second for the lifecycle, and learners choosing one should choose by which gap they are actually trying to close. A third lineage is worth naming for completeness, the serving-side build courses, which start where this pipeline ends, and the three together cover the full lifecycle from tokens to deployment.

Four mistakes account for most failed first attempts. Skipping the prerequisites assessment and stalling in step two, because the architecture assembly assumes genuine PyTorch fluency rather than acquaintance. Underestimating data-stage patience, where tokenization errors surface confusingly later in the pipeline, so the habit of validating token counts and chat formatting before training saves hours. Hardware mismatch, attempting a config above the published table and meeting the out-of-memory wall midway, when the memory flags exist precisely for that moment. And intervening on the loss curve too early, restarting runs that were progressing normally, which teaches nothing and costs the momentum that finishes pipelines. The repository’s own structure prevents most of these if followed in order, which is the strongest argument for resisting the urge to jump to the interesting steps first. If a run fails anyway, the recovery discipline is boring and effective: shrink the config until it runs, confirm each stage on the small config, then scale back up one change at a time.

As a tool, rarely; as a component, occasionally. A 13-million-parameter assistant will not compete with anything served by a frontier lab, and pretending otherwise disappoints. Where small personally-trained models genuinely work: domain-specific classifiers and extractors, where the fine-tuning data is narrow and the competition is a rule-based script rather than a large model; local prototyping, validating that a data pipeline behaves before spending on real training; and education delivery, producing a model whose every behavior you can explain, which has genuine value in teams where LLM literacy is being built. The honest mental model is that the output is a checkpoint-shaped certificate of understanding, and any incidental utility is a bonus. Teams that keep expectations calibrated this way get full value from the exercise; teams that expect a usable product have misunderstood what free-GPU scale was ever going to deliver. The sweet spot expectation is a model that demonstrates every stage convincingly on inputs you chose, which is exactly what the pipeline is engineered to produce Set that expectation on purpose, write it at the top of your notes on day one, and the build stays educational end to end rather than curdling into disappointment at step seven, when the chat window opens and a very small model answers..

Beyond convenience for terminal-shy learners, the control panel earns its place as a teaching instrument. Watching hyperparameters change behavior through a visible interface builds calibration faster than editing config files, because the feedback loop closes in seconds and the comparison between runs is laid out in front of you rather than buried in logs. It also makes the pipeline demonstrable to others, which matters in teams and classrooms, where showing a colleague the loss curve of a model you trained beats describing it. The pragmatic warning that comes with any control panel: stay in it long enough to build intuition, then graduate to the scripts, because the panel hides exactly the invocation details a real training workflow eventually requires you to know cold.