Example · 3 days
RAG and agents in production
Three days on retrieval you can measure, then agentic retrieval over mixed sources, with the production architecture to run it.
For platform and ML engineers who own retrieval systems.
Day 1
RAG as an architecture
Quarter day · RAG and agents · Intermediate
- The stages: ingest, index, retrieve, assemble, generate and cite.
- Where RAG fits, and where fine-tuning or plain search fits better.
- The failure modes of each stage.
- Choosing a vector store and a framework, or neither.
Ingestion and chunking
Quarter day · RAG and agents · Intermediate
- Parsing PDFs, HTML, Office documents and tables.
- Chunking strategies, overlap and structure-aware splitting.
- Metadata worth keeping for filtering and citations.
- Lab: ingest a corpus and inspect the worst chunks.
Embeddings and vector search
Quarter day · RAG and agents · Intermediate
- Embedding models: choosing, comparing and versioning.
- Vector indexes, similarity and approximate search.
- Filters and namespaces.
- Lab: index a corpus and run a question set against it.
Hybrid search and reranking
Quarter day · RAG and agents · Advanced
- Keyword search and where vectors lose to it.
- Hybrid retrieval and score fusion.
- Rerankers, query rewriting and generated FAQs.
- Lab: improve recall on the questions vector search misses.
Day 2
Measuring retrieval
Quarter day · RAG and agents · Intermediate
- A question set with known good sources.
- Recall, precision and ranking metrics.
- Faithfulness and answer relevance.
- Lab: compare two retrieval configurations with numbers, not impressions.
Grounded answers and citations
Quarter day · RAG and agents · Intermediate
- Assembling a grounded prompt from retrieved chunks.
- Citations a reader can follow back to the source.
- Refusing when retrieval is empty or sources conflict.
- Lab: complete the RAG path and test it on held-out questions.
Retrieval with permissions
Quarter day · RAG and agents · Advanced
- Document-level access control carried into the index.
- Tenant isolation and per-user filtering.
- Keeping permissions in sync with the source systems.
- Testing that one user cannot see another’s documents.
Running RAG in production
Quarter day · RAG and agents · Advanced
- Keeping the index fresh: incremental updates and deletions.
- Re-embedding and migrating without downtime.
- Monitoring retrieval quality and answer quality.
- Cost and latency at scale.
Day 3
Agentic retrieval
Quarter day · RAG and agents · Advanced
- Where single-shot RAG stops working.
- Query planning, decomposition and iterative retrieval.
- Knowing when to stop searching.
- Lab: turn a fixed RAG pipeline into a retrieval agent.
Retrieval over mixed sources
Quarter day · RAG and agents · Advanced
- Documents, databases and APIs behind one question.
- Text-to-SQL, and the guardrails it needs.
- Routing questions to the right source.
- Combining results into one grounded answer.
Agent failures, permissions and human approval
Quarter day · RAG and agents · Advanced
- How agents fail: loops, drift and confident wrong actions.
- Permission boundaries and scoped credentials.
- Human approval at the decisions that matter.
- Lab: add approvals and limits to a working agent.
Evaluating agents and multi-step systems
Quarter day · AI engineering · Advanced
- Scoring the outcome against scoring the path taken.
- Tool-call accuracy, step counts and recovery from errors.
- Simulated users and environments.
- Lab: evaluate an agent on a task set and find its weakest step.