Duration
2 days · Technical · Introductory
Skills Gained
-
Call models from code with structured output and bounded retries, connect a tool through function calling and MCP, and explain where LLMs fail
-
Write a specification for an LLM feature whose contracts, limits and acceptance criteria drive the prompts, the code and the tests
-
Ground answers in a document set with lean, cached context, hybrid search and citations
-
Turn acceptance criteria into a test set, and trace quality, cost and latency for every change
-
Test for prompt injection and data leakage, and add guardrails before release
-
Take a new feature from spec to a tested, demo-ready release on your own
Who Can Benefit
-
Software developers who want hands-on practice with the latest AI capabilities and the practices teams use to ship them
-
Tech leads and architects setting how their teams specify, build and review LLM features
-
Data engineers who supply the documents and data these features run on
Prerequisites
Organizational Objectives
-
LLM features that meet a written spec and pass a test set before they ship
-
Fewer AI pilots stalled between demo and production
-
Cost, security and permissions handled from the first build
-
One shared pattern for specifying, building and reviewing LLM features across teams
Software
A web browser and a provided cloud lab with Python, model API access, an open-weight model as a fallback, Postgres with pgvector, and an open-source evaluation tool. No special licenses needed.
Day 1 · Specify and build.
Module 1: Model Access and Tool Basics
By the end of this module, you will be able to call models from code, get structured output, connect a tool, and know where models fail.
-
Today's hosted and open-weight models, and when a reasoning model is worth the cost
-
Structured output with schemas and validation; streaming, timeouts and bounded retries
-
Function calling, and exposing a tool through MCP
-
Agentic workflows: a fixed workflow or an agent, and one tool loop with a step limit
-
LLM limitations: hallucination, knowledge cutoffs, non-determinism and context limits
-
Hands-on Lab: Run the same task on two models with validated output, then add a ticket tool the assistant calls only after a person approves.
Module 2: Spec-Driven Development
By the end of this module, you will be able to write a specification that the prompts, the code and the tests all follow.
-
Why prompt-first features drift, and the spec as the single source of truth
-
What an LLM feature spec holds: purpose, inputs and outputs as schemas, tool permissions, refusal behavior and acceptance criteria
-
Writing acceptance criteria a test can check
-
Keeping the spec in the repository, versioned and reviewed with the code, and working from it with AI coding assistants
-
Hands-on Lab: Write the spec for the document assistant, turn its acceptance criteria into the first 10 test cases, and check the current build against them.
Module 3: Retrieval and Context Engineering
By the end of this module, you will be able to give the model the right instructions and documents, and nothing more.
-
System instructions, examples and what earns a place in the context window
-
Prompt caching: the unchanging part of the prompt goes first
-
Checking document quality, then chunking, embeddings, and hybrid keyword and vector search
-
Citations, reranking, answering "I don't know", and respecting document permissions
-
When long context can replace retrieval
-
Hands-on Lab: Index a document set, compare vector-only and hybrid search on how often the right passage is found, and trim the context while the spec's test cases still pass.
Day 2 · Prove it, protect it, ship it.
Module 4: Evaluation, Cost, and Observability
By the end of this module, you will be able to show the assistant meets its spec, what it costs, and what each request did.
-
Reading real failures before writing more tests
-
A test set traced to the spec: rule-based checks first, then a model as judge, checked against human labels
-
One trace per request: tokens, cost and latency
-
Spend limits, caching savings, and sending simple requests to smaller models
-
A release checklist: pinned model versions, passing test set, known costs
-
Hands-on Lab: Grow the spec's test cases into a 40-case set, find a failure, fix it in the spec and the build, and show the score and cost per request in the trace log.
Module 5: Guardrails and Security
By the end of this module, you will be able to test the assistant against misuse and limit what it can do.
-
Prompt injection, typed by a user or hidden in documents and tool results
-
Data leakage, and the assistant doing more than it was asked
-
Security requirements in the spec: what the assistant must never do
-
Input and output guardrails, least privilege, and approval enforced in your application
-
Hands-on Lab: Plant an instruction in a document that tries to file a ticket, show that it works, add a defense, and add the attack to the spec and the test set.
Module 6: Capstone: From Spec to Release
By the end of this module, you will be able to take a new LLM feature from spec to a tested release on your own.
-
A new feature brief, such as summarizing a support ticket thread
-
Writing its spec and acceptance criteria
-
Building it on the existing assistant with structured output and retrieval
-
Turning the criteria into test cases, plus one injection test
-
The release checklist: test score, cost per request, pinned model version
-
Hands-on Lab: Ship the feature in about 90 minutes, then demo its spec, test score and cost to the group.