AI Engineering Foundations

Course code: PT1024

From Spec to a Tested LLM Feature Overview

Build products with AI that your team can trust.

AI engineering is harnessing the power of modern LLM models using disciplined engineering workflows to deliver quality and stability in production workflows. Over two days, you start each feature from a written specification, then build to it with the latest model capabilities: structured output, tool calling, the Model Context Protocol (MCP), retrieval and prompt caching. You learn the practices leading teams use to test, secure and control the cost of what they build, and finish with one working assistant that answers questions from a set of documents, takes one action, and passes a test suite drawn from its spec.

Request a Consultant

About This Course

Duration

2 days · Technical · Introductory

Skills Gained

  • Call models from code with structured output and bounded retries, connect a tool through function calling and MCP, and explain where LLMs fail

  • Write a specification for an LLM feature whose contracts, limits and acceptance criteria drive the prompts, the code and the tests

  • Ground answers in a document set with lean, cached context, hybrid search and citations

  • Turn acceptance criteria into a test set, and trace quality, cost and latency for every change

  • Test for prompt injection and data leakage, and add guardrails before release

  • Take a new feature from spec to a tested, demo-ready release on your own

Who Can Benefit

  • Software developers who want hands-on practice with the latest AI capabilities and the practices teams use to ship them

  • Tech leads and architects setting how their teams specify, build and review LLM features

  • Data engineers who supply the documents and data these features run on

Prerequisites

  • Working Python

  • REST APIs and JSON

  • Git

  • No machine learning background needed

Organizational Objectives

  • LLM features that meet a written spec and pass a test set before they ship

  • Fewer AI pilots stalled between demo and production

  • Cost, security and permissions handled from the first build

  • One shared pattern for specifying, building and reviewing LLM features across teams

Software

A web browser and a provided cloud lab with Python, model API access, an open-weight model as a fallback, Postgres with pgvector, and an open-source evaluation tool. No special licenses needed.

You'll Learn

Day 1 · Specify and build.

Module 1: Model Access and Tool Basics

By the end of this module, you will be able to call models from code, get structured output, connect a tool, and know where models fail.

  • Today's hosted and open-weight models, and when a reasoning model is worth the cost

  • Structured output with schemas and validation; streaming, timeouts and bounded retries

  • Function calling, and exposing a tool through MCP

  • Agentic workflows: a fixed workflow or an agent, and one tool loop with a step limit

  • LLM limitations: hallucination, knowledge cutoffs, non-determinism and context limits

  • Hands-on Lab: Run the same task on two models with validated output, then add a ticket tool the assistant calls only after a person approves.

Module 2: Spec-Driven Development

By the end of this module, you will be able to write a specification that the prompts, the code and the tests all follow.

  • Why prompt-first features drift, and the spec as the single source of truth

  • What an LLM feature spec holds: purpose, inputs and outputs as schemas, tool permissions, refusal behavior and acceptance criteria

  • Writing acceptance criteria a test can check

  • Keeping the spec in the repository, versioned and reviewed with the code, and working from it with AI coding assistants

  • Hands-on Lab: Write the spec for the document assistant, turn its acceptance criteria into the first 10 test cases, and check the current build against them.

Module 3: Retrieval and Context Engineering

By the end of this module, you will be able to give the model the right instructions and documents, and nothing more.

  • System instructions, examples and what earns a place in the context window

  • Prompt caching: the unchanging part of the prompt goes first

  • Checking document quality, then chunking, embeddings, and hybrid keyword and vector search

  • Citations, reranking, answering "I don't know", and respecting document permissions

  • When long context can replace retrieval

  • Hands-on Lab: Index a document set, compare vector-only and hybrid search on how often the right passage is found, and trim the context while the spec's test cases still pass.

Day 2 · Prove it, protect it, ship it.

Module 4: Evaluation, Cost, and Observability

By the end of this module, you will be able to show the assistant meets its spec, what it costs, and what each request did.

  • Reading real failures before writing more tests

  • A test set traced to the spec: rule-based checks first, then a model as judge, checked against human labels

  • One trace per request: tokens, cost and latency

  • Spend limits, caching savings, and sending simple requests to smaller models

  • A release checklist: pinned model versions, passing test set, known costs

  • Hands-on Lab: Grow the spec's test cases into a 40-case set, find a failure, fix it in the spec and the build, and show the score and cost per request in the trace log.

Module 5: Guardrails and Security

By the end of this module, you will be able to test the assistant against misuse and limit what it can do.

  • Prompt injection, typed by a user or hidden in documents and tool results

  • Data leakage, and the assistant doing more than it was asked

  • Security requirements in the spec: what the assistant must never do

  • Input and output guardrails, least privilege, and approval enforced in your application

  • Hands-on Lab: Plant an instruction in a document that tries to file a ticket, show that it works, add a defense, and add the attack to the spec and the test set.

Module 6: Capstone: From Spec to Release

By the end of this module, you will be able to take a new LLM feature from spec to a tested release on your own.

  • A new feature brief, such as summarizing a support ticket thread

  • Writing its spec and acceptance criteria

  • Building it on the existing assistant with structured output and retrieval

  • Turning the criteria into test cases, plus one injection test

  • The release checklist: test score, cost per request, pinned model version

  • Hands-on Lab: Ship the feature in about 90 minutes, then demo its spec, test score and cost to the group.

Request a Consultant

Speak with a Learning Consultant