BuildLLM

What you will learn You will download BuildLLM, create the one Python environment used by the entire project, install the correct PyTorch build for your hardware, and verify that Python can see the CPU or GPU you intend to use. Where you are in the build Nothing has been trained yet. The goal of this…

What you will learn You will build a practical mental model of artificial intelligence, machine learning, neural networks, language models, pretraining, inference, tokens, parameters, context, loss, epochs, and hallucination. These terms will appear in every later lesson. Where you are in the build Your repository and Python environment are ready. Before downloading data, you need…

What you will learn You will download and extract Simple Wikipedia into a local JSON Lines file, inspect the extraction report, read several accepted documents, and verify that the first source required by the corpus builder is ready. Where you are in the build The model will not train directly from the Wikipedia dump. This…

What you will learn You will combine source documents into a cleaned, filtered, deduplicated, topic-balanced JSONL corpus. You will learn what each corpus configuration controls, why the 50M profile is the recommended learning path, how to inspect the report, and when to choose the pipeline-verification, balanced, or 800M profiles. Where you are in the build…

What you will learn You will train the tokenizer that defines the model’s vocabulary, convert the selected corpus into integer token IDs, split complete documents into training and validation regions, and understand how those token regions become next-token examples. Where you are in the build The transformer does not receive raw text. It receives token…

What you will learn In this lesson you will follow one sequence of token IDs through the decoder-only transformer until the model produces a score for every possible next token. You will learn what embeddings, causal self-attention, attention heads, feed-forward networks, normalization, residual connections, logits, and weight tying contribute to that process. Where you are…

What you will learn In this lesson you will follow one optimizer step from a batch of token IDs to an update of the model’s weights. You will learn how loss, backpropagation, AdamW, learning-rate scheduling, gradient clipping, mixed precision, validation, and checkpoint decisions work together. Where you are in the build The model can already…

What you will learn In this lesson you will verify the complete training path, run the first meaningful CPU learning experiment, inspect its evidence, and decide whether the result is healthy enough to evaluate. Iteration 1 proves that the same framework used by the larger models works from corpus to checkpoint. Where you are in…

What you will learn In this lesson you will load the Iteration 1 checkpoint with its matching tokenizer, turn logits into generated tokens, compare decoding settings, and evaluate the model with more evidence than one impressive or disappointing sample. Where you are in the build Training changed the model’s weights. Generation is the first time…

What you will learn In this lesson you will learn how the same BuildLLM model and trainer run on a central processing unit (CPU) or a CUDA-capable graphics processing unit (GPU), how numeric precision affects speed and memory, how to respond to out-of-memory errors, and how to estimate a long run from measured throughput. Where…

What you will learn In this lesson you will train the 42.1M-parameter model in two distinct experiments. First, you will use the same 50M corpus, tokenizer, example limits, and epoch count as Iteration 1 to make a controlled comparison. Then you will build the broader balanced corpus and run the intended larger-data training path for…

What you will learn In this lesson you will learn what BuildLLM saves, how atomic checkpoint writes reduce corruption risk, how –resume selects and restores state, what is not restored inside an interrupted epoch, and how to perform a safe recovery exercise. Where you are in the build A long run is only useful if…

What you will learn In this lesson you will work with the approximately 100M-parameter reference model as a long-running systems-engineering project. You will plan and run a controlled 50M experiment, prepare the complete 800M data path, understand the eighteen-epoch run, monitor useful evidence, interrupt and resume safely, and generate with the correct tokenizer. Where you…

What you will learn In this lesson you will turn run records, validation measurements, fixed-prompt samples, and operational evidence into defensible conclusions. You will create one table for the controlled 50M comparison and a separate table for the larger-data training path. Where you are in the build Comparison is not a victory lap for the…

Use this glossary when a course term is unfamiliar. Definitions describe how the term is used in this repository. Repository and data artifacts Term Plain-language meaning Repository The project directory containing code, configurations, documentation, and tests. Virtual environment The .venv directory that isolates this project’s Python packages from other projects. Dependency A software package the…

Copy this worksheet before every training experiment. Fill values from configuration files, reports, startup JSON, epoch output, final JSON, checkpoints, and generated samples. Use <not run> or <not available> instead of leaving an ambiguous blank. Experiment question What specific question is this run intended to answer? <your question> Experiment path: <controlled comparison / larger-data training…

Start with the stage that failed. Preserve the full command and error before changing anything. Make one correction at a time so you know which change solved the problem. Environment and repository Symptom Check Response python or py is not recognized Python is not installed or not on the shell path Install a supported Python…

This page shows where each course topic appears in the code. Iteration 3 uses this complete path. Iterations 1 and 2 use the same path with different model profiles. End-to-end call path Configuration File Role configs/models/*.json Versioned iteration dimensions and runtime defaults src/model_profiles.py Validates profile JSON and selects CPU/GPU defaults src/config.py Complete serializable ModelConfig used…