Engineering Generative AI Applications: Build, Evaluate, Deploy, and Scale

Duration

4 days

Skills Gained

By the end of this course, participants will be able to:

• Build conversational and content-generation features against hosted LLM APIs

• Compose LangChain pipelines for prompt-templated, structured, and RAG-based applications

• Evaluate LLM applications using metrics, custom criteria, and DeepEval/G-Eval

• Configure deployment, scaling, and monitoring strategies for GenAI in production

• Apply LangGraph patterns to build basic agentic LLM applications

Who Can Benefit

This course is designed for:

• Software Developers

• Data Scientists & Analysts

Prerequisites

Participants should enter this course with:

• Familiarity using Generative AI

• Practical Python experience

• Recommended (not required): Mastering Prompt Engineering, or equivalent prompt engineering experience

Organizational Objectives

This course assists organizations to:

• Accelerate GenAI feature delivery through reusable LangChain pipelines and prompt templates

• Reduce production incidents by codifying monitoring metrics, alert thresholds, and rollback procedures

• Lower agent error rates through evaluation pipelines that catch regressions before deploy

• Build a working knowledge of agentic AI patterns across the development team in four days

Software

All attendees must have a modern web browser and an Internet connection.

Module 1 - Understanding Generative AI (Optional Recap)

This module is an optional recap for participants who want a refresher before Module 2. By the end of it, you will be able to describe how foundation models work at a high level and recognize the model settings that influence output quality.

• Generative AI vs. traditional AI, and how foundation models work (tokenization, embeddings, decoding)

• LLM settings and parameters; intro to prompt engineering, RAG, and agentic AI

• Hands-on Lab: Explore tokenization, embeddings, and core LLM settings hands-on without writing any code.

Module 2 - Basics of Prompt Engineering

By the end of this module, you will be able to write structured prompts using the CLEAR and RCTO/RTF frameworks, iterate prompts through a deliberate loop, and decompose tasks into sequenced prompts with human checkpoints between stages.

• CLEAR framework

• RCTO / RTF structure

• Adaptive iteration loop

• Workflow decomposition (analyze/transform/write)

• Human checkpoints between stages

• Prompt-vs-workflow-vs-tool decision

• Hands-on Lab: Author and iterate one effective ChatGPT prompt that uses CLEAR and at least one n-shot variant for a workplace task.

Module 3 - Building Simple LLM-Based Applications

By the end of this module, you will be able to call LLMs through hosted APIs, structure prompts with prompt templates, build a conversational completion loop, and choose between closed-weight and open-weight models for cost and latency tradeoffs.

• Application design building blocks

• Accessing LLMs via APIs

• Prompt templates

• Conversational model of completion

• Closed-weight vs. open-weight models

• Batch APIs for cost control

• Hands-on Lab: Build a vanilla chatbot against the OpenAI API using a prompt template and a conversational completion loop.

Module 4 - Accelerating Development with LangChain

By the end of this module, you will be able to compose LangChain primitives into a working application, produce structured output through function calling, and assemble RAG pipelines from loaders, splitters, parsers, and a vector retriever.

• LangChain terminology and conceptual overview

• LLM chains and prompt templates

• Loaders, splitters, and parsers

• Retrieval Augmented Generation (RAG)

• Structured output and function calling

• Composing chains end-to-end for production tasks

• Hands-on Lab: Build a LangChain pipeline that returns structured output from a prompt-chained sequence of calls.

Module 5 - Basics of Evaluating Generative AI Applications

By the end of this module, you will be able to apply automated metrics to GenAI output, distinguish GenAI evaluation from conventional software testing, interpret evaluation results, and define custom evaluation metrics for application-specific quality.

• Generative AI evaluation vs. software evaluation

• Metrics for evaluating generative AI

• Interpreting evaluation results

• Advanced metrics and techniques for generative AI

• Custom evaluation metrics

• Hands-on Lab: Use DeepEval and G-Eval to evaluate one LLM application against built-in and custom criteria.

Module 6 - Deploying and Scaling Generative AI Applications

By the end of this module, you will be able to recognize the unique deployment challenges generative AI applications introduce, manage latency and throughput at scale, choose between cloud and on-premise deployments, and operate continuous-deployment pipelines with monitoring and auto-scaling.

• Unique deployment and scaling challenges for generative AI

• Managing latency and throughput

• Ensuring data privacy and security

• Cloud vs. on-premise deployment tradeoffs

• Continuous deployment pipelines for GenAI

• Monitoring and auto-scaling GenAI applications

• Hands-on Lab: Apply scaling laws and a batching/caching strategy to optimize one GenAI deployment.

Module 7 - Basics of Generative AI Monitoring

By the end of this module, you will be able to distinguish ongoing monitoring from offline evaluation, choose key monitoring metrics for an LLM application in production, instrument an alert and log pipeline, and verify the monitoring system end-to-end.

• Differences Between Evaluation and Monitoring

• Identifying Key Monitoring Metrics

• Understanding the Monitoring Workflow

• Alerts, Logs, and Monitoring Verification

• Setting Up a Monitoring System

• Hands-on Lab: Stand up a monitoring system for one GenAI application with key metrics, alert thresholds, and verification.

Module 8 - Debugging and Testing Generative AI Applications

By the end of this module, you will be able to instrument GenAI applications for debugging, write tests that survive non-determinism, apply debugging and testing tools and strategies appropriate to LLM systems, and integrate them into CI/CD pipelines.

• Debugging generative AI applications

• Testing generative AI applications

• Debugging and testing tools

• Debugging and testing strategies

• Continuous integration and deployment for GenAI

• Hands-on Lab: Debug a chatbot application and add a test harness covering the failure modes you uncovered.

Module 9 - Basics of Agentic AI with LangGraph

By the end of this module, you will be able to build agentic LLM applications with LangGraph, recognize common agentic patterns, coordinate multi-agent workflows, and handle errors that arise when agents call tools and each other.

• LangGraph concepts and principles

• Common agentic patterns

• Multi-agent workflows

• Agent communication and coordination

• Error handling in agentic systems

• Scaling agentic AI applications

• Hands-on Lab: Build an agentic router and a ReAct agent with LangGraph that share a tool and coordinate via a shared state graph.

Module 10 - Generative AI Use Cases and Ethics

By the end of this module, you will be able to identify high-value generative AI use cases inside your work context, recognize the ethical considerations that govern responsible use, and reason about AI-related risk through real case studies.

• Real-world applications of generative AI

• Ethical considerations in generative AI

• Responsible use of generative AI

• Case studies and examples

• Hands-on Lab: Identify candidate GenAI use cases in your team’s workflow and spot the ethical issues each one raises.