Ready-made course · 2 days · Advanced
Transformers from First Principles
A two-day course for technically strong practitioners: build a usable understanding of transformer models by implementing simplified components, without unnecessary mathematical formalism.
Transformer models sit under almost every production language system in current use. Many engineers can call an API without being able to reason about attention, context, or why scaling and quantisation behave as they do. This course closes that gap by building the mechanism from the pieces.
The course starts with vectors, embeddings and matrix operations at an intuitive level, then implements attention, self-attention, queries, keys and values. Positional information, transformer blocks, feed-forward layers and normalisation are assembled into a working picture of a model. Training, loss, token prediction and inference are connected to that implementation.
The mathematics is kept as light as the subject allows. You write code for simplified transformer components and inspect them, rather than being asked to reproduce a paper. Scaling and production implications are drawn out as the implementation grows.
You leave able to read a transformer architecture, discuss it with ML specialists, and make better choices about context, fine-tuning and local inference.
- Who it is for
- Developers, ML engineers and technically strong practitioners who want a deeper, implementation-level understanding of transformer models.
- Before you start
- Comfortable writing code and with high-school algebra. Some familiarity with arrays or tensors helps. No requirement for formal machine-learning theory or advanced calculus.
You will be able to
- Explain embeddings, attention and transformer blocks in implementation terms.
- Implement simplified self-attention, including queries, keys, values and positional information.
- Assemble a transformer block: attention, feed-forward layers, residual connections and normalisation.
- Connect training, loss and token prediction to the forward pass the class has built.
- Describe inference, scaling and the practical limits that follow from the architecture.
- Use the implementation as a basis for better decisions about context, fine-tuning and runtime.
Modules
8 quarter-day modules. Each one can go into a custom course on its own.
Tokenisation and embeddings from scratch
- Byte-pair encoding, implemented.
- Embedding tables and positional information.
- What the representation makes easy and hard.
- Lab: build and train a tokeniser.
Self-attention from scratch
- Queries, keys and values as matrix operations.
- Scaled dot-product attention and causal masking.
- Multi-head attention.
- Lab: implement attention in PyTorch and check it by hand.
The transformer block
- Residual connections, normalisation and feed-forward layers.
- Stacking blocks into a decoder-only model.
- Modern variants: RoPE, grouped-query attention and mixture of experts.
- Lab: assemble a small GPT.
The pretraining objective and training loop
- Cross-entropy on the next token.
- Optimisers, learning-rate schedules and batching.
- Watching loss curves and catching problems early.
- Lab: train the small GPT on a real corpus.
Inference, the KV cache and decoding
- The KV cache and why generation is memory-bound.
- Greedy, sampling and speculative decoding.
- Batching and throughput on a GPU.
- Lab: add a KV cache and measure the speed-up.
Scale, compute and scaling laws
- Parameters, data and compute, and how they trade off.
- Scaling laws and what they predict.
- Why frontier training is a different engineering problem.
- Small models that punch above their size.
Evaluating a base model
- Perplexity and its limits.
- Benchmarks, contamination and what they really measure.
- Task-specific evaluation for your own domain.
- Lab: evaluate checkpoints and pick one.
Lab: a small language model, end to end
- Put the pieces together: tokeniser, model, training and sampling.
- Train on a domain corpus.
- Compare outputs at different stages of training.
- Write up what the model learned, and what it did not.
Practical work
You implement simplified transformer components in code: embeddings, attention, a block, and a tiny training and generation loop. The objective is understanding, not reproducing a production-scale model.
You take away
- Working code for simplified transformer components
- Visualisations of attention and token prediction
- A compact architecture reference for production discussions
- A clearer map from internals to context, fine-tuning and inference choices
Combine it with
- Fine-Tuning Language Models · 2 days
- Training Your Own Language Model · 3 days
- Running LLMs Locally · 1 day