Data and Evaluation Driven Development of AI Agents

About this session

In this practical, step-by-step session, we’ll dive deep into the data and evaluation-driven development process of AI agents using a compelling real-world scenario. Participants will start by defining clear business objectives and translating these into targeted evaluation metrics critical for agent success. Next, we’ll collaboratively generate and curate high-quality synthetic data designed specifically to meet these business-driven metrics. Using this dataset, attendees will set up structured evaluations, emphasizing real-world performance indicators like accuracy, reliability, and alignment with business KPIs. With our data and metrics firmly established, we’ll move to hands-on development, leveraging popular frameworks to build, evaluate, and refine an actual AI agent live. Participants will gain practical insights into common pitfalls and learn best practices for iterative agent improvement using continuous evaluation feedback. By session end, attendees will have experienced the full lifecycle- planning, data generation, evaluation setup, development, and optimization thereby equipping them with actionable strategies and tangible skills to immediately implement in their own AI agent projects.