Skip to content
Linear Horizon Training

Ready-made course · 1 day · Intermediate

Running LLMs Locally

A one-day practical course on local and private inference: model formats, hardware, quantisation, engines, benchmarking and the privacy and cost reasons to run models yourself.

Local inference is no longer only a hobbyist concern. Teams run models on laptops, on-prem GPUs and private cloud for privacy, cost, latency or air-gapped work. The difficulty is operational: choosing a model size that fits VRAM, a format the engine can load, and a configuration that meets a latency target.

This course is practical. You work through model families, formats, sizes, quantisation, inference engines, CPU versus GPU, throughput, latency, context length and batching. You expose a local API, measure it, and discuss hardware selection and deployment.

Privacy and cost are treated as the usual reasons to do this work, not as slogans. The course is honest about quality loss from aggressive quantisation and about the gap between a laptop demo and a multi-user service.

You leave able to select, run, benchmark and expose a local model, and to decide when local inference is the right deployment rather than a preferred ideology.

Who it is for
Developers, infrastructure engineers and AI engineers who need to run or support local and private inference.
Before you start
Comfortable with the command line and with installing local tooling. A machine with enough RAM, and a GPU if available, makes the labs more realistic. No ML research background is required.

You will be able to

  • Select a model family, size and quantisation that fit a given hardware envelope.
  • Run local inference with a current engine and expose it behind a local API.
  • Measure latency, throughput, context length and quality trade-offs.
  • Compare CPU and GPU paths, including batching and concurrent users.
  • Design a private deployment that matches privacy, cost and operational constraints.
  • Choose hardware with a defensible view of VRAM, context and expected load.

Modules

4 quarter-day modules. Each one can go into a custom course on its own.

  1. Open models, formats and hardware

    • The open-weight model landscape and licences.
    • Quantisation and formats: GGUF, AWQ and others.
    • Sizing hardware: memory, GPUs and Apple silicon.
    • Lab: run the same model at different quantisation levels.
  2. Serving local models

    • Ollama and llama.cpp on a laptop or workstation.
    • vLLM and other servers for shared use.
    • OpenAI-compatible APIs, so applications do not change.
    • Lab: serve a model and call it from an application.
  3. Local models for confidential data

    • When data cannot leave your network.
    • Private RAG on local models.
    • Where local models fall short, and hybrid patterns.
    • Security for a self-hosted model service.
  4. Benchmarking and deploying local models

    • Measuring quality on your own tasks.
    • Throughput, latency and cost per request.
    • Packaging and deployment on servers or Kubernetes.
    • Lab: benchmark two models and recommend one.

Practical work

You install or use a local inference engine, load a quantised model, expose an API, and benchmark latency and throughput. You compare configurations and produce a short hardware and deployment recommendation.

You take away

  • A running local model and API
  • Benchmark notes for latency, throughput and quantisation
  • A hardware and model-selection checklist
  • Deployment options for private inference

Combine it with

Contact