Ready-made course · 3 days · Advanced
Training Your Own Language Model
A three-day course for advanced developers and ML engineers: build and train a small language model in order to understand the process, not to compete with frontier systems.
Training a language model looks opaque from the outside. This course demystifies it by having you construct a dataset, tokenise, implement a small transformer, and run a training loop you can inspect. The objective is understanding. Nobody leaves with a model that replaces a production API, and that is stated at the start.
The course covers dataset construction, tokenisation, embeddings, architecture, the training loop, loss, optimisation, batching, validation, checkpoints and inference. GPU constraints, scaling and evaluation are discussed as the reasons large training runs look the way they do.
You train a small working model. You watch loss move, inspect generations, and connect those observations to the production models you already call.
The value is a team that can talk to ML engineers, judge training proposals, and stop treating “we should train our own model” as either magic or a strategy.
- Who it is for
- Advanced developers and ML engineers who need a grounded understanding of language-model training.
- Before you start
- Strong Python, comfort with the transformer course or equivalent, and access to a GPU environment. This is not a first course in AI.
You will be able to
- Construct a small training dataset and a tokeniser appropriate to the experiment.
- Implement or assemble a small transformer and explain each stage of the forward pass.
- Run a training loop with batching, loss, optimisation, validation and checkpoints.
- Generate from a checkpoint and evaluate it honestly against the training objective.
- Relate GPU memory, batch size and context length to what is feasible in the course and in a real cluster.
- Judge when training from scratch is the wrong proposal compared with adapting or calling an existing model.
Modules
12 quarter-day modules. Each one can go into a custom course on its own.
Building a training corpus
- Sources, licences and provenance.
- Cleaning, deduplication and quality filtering.
- Mixtures and domain weighting.
- Lab: build and profile a small corpus.
Tokenisation and embeddings from scratch
- Byte-pair encoding, implemented.
- Embedding tables and positional information.
- What the representation makes easy and hard.
- Lab: build and train a tokeniser.
Self-attention from scratch
- Queries, keys and values as matrix operations.
- Scaled dot-product attention and causal masking.
- Multi-head attention.
- Lab: implement attention in PyTorch and check it by hand.
The transformer block
- Residual connections, normalisation and feed-forward layers.
- Stacking blocks into a decoder-only model.
- Modern variants: RoPE, grouped-query attention and mixture of experts.
- Lab: assemble a small GPT.
The pretraining objective and training loop
- Cross-entropy on the next token.
- Optimisers, learning-rate schedules and batching.
- Watching loss curves and catching problems early.
- Lab: train the small GPT on a real corpus.
Training at scale
- Mixed precision and memory budgets.
- Data, tensor and pipeline parallelism.
- Checkpointing and restarts.
- Estimating cost and time before you start.
Diagnosing a training run
- Loss spikes, divergence and dead runs.
- Gradient norms, learning rates and other signals.
- Ablations that answer one question at a time.
- Lab: find the fault in a broken run.
Scale, compute and scaling laws
- Parameters, data and compute, and how they trade off.
- Scaling laws and what they predict.
- Why frontier training is a different engineering problem.
- Small models that punch above their size.
Inference, the KV cache and decoding
- The KV cache and why generation is memory-bound.
- Greedy, sampling and speculative decoding.
- Batching and throughput on a GPU.
- Lab: add a KV cache and measure the speed-up.
Evaluating a base model
- Perplexity and its limits.
- Benchmarks, contamination and what they really measure.
- Task-specific evaluation for your own domain.
- Lab: evaluate checkpoints and pick one.
Whether to fine-tune
- What fine-tuning changes, and what it cannot fix.
- Prompting, retrieval and fine-tuning compared on real problems.
- Cost, data and maintenance.
- Exercise: decide for three candidate use cases.
Lab: a small language model, end to end
- Put the pieces together: tokeniser, model, training and sampling.
- Train on a domain corpus.
- Compare outputs at different stages of training.
- Write up what the model learned, and what it did not.
Practical work
You train a small working language model. You own the dataset, the loop, the checkpoints and the evaluation. The closing artefact is the model plus a short report that states what was learned and what would not transfer to a production training programme.
You take away
- A small trained model and checkpoints
- A training-loop implementation and run notes
- An evaluation report with an honest scope
- A framework for judging training proposals in an organisation
Combine it with
- Transformers from First Principles · 2 days
- Fine-Tuning Language Models · 2 days
- Running LLMs Locally · 1 day