Skip to content
Linear Horizon Training

Ready-made course · 1 day · Intermediate

Evaluating and Testing LLM Applications

A one-day course on evaluating non-deterministic systems: golden datasets, semantic checks, LLM-as-judge, retrieval and agent evaluation, and the pipeline that keeps quality from drifting.

Conventional software testing assumes that the same input should produce the same output. LLM applications violate that assumption while still needing regression control, release judgement and production monitoring. Teams that skip this work discover quality issues only through users.

This course teaches a practical evaluation discipline. You separate deterministic tests from semantic evaluation, build golden datasets, and use human review and LLM-as-judge where they are justified. Retrieval is evaluated separately from generation. Groundedness, hallucination, agent behaviour and prompt or model changes are treated as first-class concerns.

The day is a build. You construct an evaluation suite around a realistic application path, then look at how that suite becomes a pipeline: versioning, model comparison, regression gates and production monitoring.

You leave able to say whether a change made the system better, worse, or merely different.

Who it is for
AI developers, QA engineers, ML engineers and technical leads responsible for quality, release and monitoring of LLM applications.
Before you start
Experience building or testing software. Familiarity with LLM applications is expected. You do not need a research background in evaluation metrics.

You will be able to

  • Explain why conventional tests are necessary but insufficient for LLM applications.
  • Design golden datasets that cover the behaviours the organisation actually cares about.
  • Implement deterministic checks, semantic evaluation and targeted human review.
  • Evaluate retrieval, groundedness and hallucination separately from fluency.
  • Compare prompts and models with a regression set rather than with anecdotes.
  • Evaluate agent traces: tool choice, loops, permissions and stopping conditions.
  • Design an evaluation pipeline that can run in development and in production monitoring.

Modules

4 quarter-day modules. Each one can go into a custom course on its own.

  1. What “correct” means for an LLM system

    • Non-determinism, and why one good answer proves little.
    • Turning product requirements into criteria you can score.
    • Exact checks, semantic checks and human judgement.
    • Exercise: write the acceptance criteria for a real feature.
  2. Golden datasets and evaluators

    • Building a golden set from real traffic and expert input.
    • Code-based evaluators and LLM-as-judge.
    • Calibrating a judge against human labels.
    • Lab: build a golden set and score two prompt versions.
  3. Evaluating agents and multi-step systems

    • Scoring the outcome against scoring the path taken.
    • Tool-call accuracy, step counts and recovery from errors.
    • Simulated users and environments.
    • Lab: evaluate an agent on a task set and find its weakest step.
  4. Evaluation in CI and in production

    • Regression suites that run when a prompt or model changes.
    • Online evaluation, sampling and user feedback.
    • Drift, alerts and the dashboard someone actually reads.
    • Lab: wire evaluations into a pipeline.

Practical work

You build an evaluation suite around a representative application: a golden set, mixed checkers, retrieval and generation scores, and a simple regression report. The closing work is a pipeline design that could be implemented on your own system.

You take away

  • A working evaluation suite
  • Templates for golden datasets and score definitions
  • A regression report format for prompt and model changes
  • A pipeline sketch for CI and production monitoring

Combine it with

Contact