Evaluation and testing
Evaluation for LLM Applications
Learn LLM evaluation through error analysis, observability, evaluation techniques and monitoring.
A small sample of 30,000+ courses curated from around the world, all available to DoJo members.
A free 30-minute conversation with Ted about your goals.
Already a member? Open this course (opens in a new tab)
Course topics
Evaluation and testing
- Error analysis
- Evaluation techniques
- Production monitoring
You inspect the evidence
Find your fit
Is this a useful starting point?
For learners examining LLM quality; basic AI or software-development knowledge is helpful but not required.
- Starting point
- Check requirements
- Course length
- 1 total hour
- Lessons
- 24 lectures
Course by Digital Innovation | Les Experts
Course last updated: 09/2025
Requirements
- No strict prerequisites — basic knowledge of AI or software development is helpful but not required.
What is included
- 1 hour on-demand video
- 1 article
- 1 downloadable resource
- Access on mobile
- Certificate of completion
DoJo membership: A$250 per month per seat, excluding GST. Includes fortnightly group support and a private member community.
Explore the lessons
Course curriculum.
9 sections • 24 lectures • 59m total length
Introduction2 lectures • 4min
- Introduction03:57
- Download Course Materials00:00
Section 1: Foundations of LLM Evaluation3 lectures • 7min
- Types of evaluations – intrinsic vs extrinsic02:23
- What makes an LLM "good"? (accuracy, helpfulness, safety, latency)02:30
- Challenges in evaluating generative outputs02:34
- Quiz – Section 1: Foundations of LLM Evaluation 5 questions
Section 2: Instrumentation & Observability3 lectures • 8min
- Logging LLM inputs, outputs, and metadata02:36
- Setting up observability pipelines (OpenTelemetry, Prometheus, etc.)02:23
- Metrics to track (latency, token usage, user satisfaction)02:33
- Quiz – Section 2: Instrumentation & Observability 5 questions
Section 3: Systematic Error Analysis3 lectures • 7min
- Categorizing LLM failures (hallucinations, bias, toxicity)02:23
- Root cause analysis frameworks02:28
- Feedback loops and error logging strategies02:28
- Quiz – Section 3: Systematic Error Analysis 5 questions
Section 4: Evaluation Techniques & LLM-Judge Approaches3 lectures • 7min
- Human evaluation vs automatic evaluation02:17
- Using LLMs to grade other LLMs (LLM-as-a-judge techniques)02:35
- Pairwise comparison and scoring methods02:21
- Quiz – Section 4: Evaluation Techniques & LLM-Judge Approaches 5 questions
Section 5: Evaluating RAG Systems3 lectures • 7min
- What makes Retrieval-Augmented Generation different?02:16
- Evaluating retrieval quality (recall, precision, relevance)02:22
- Combined evaluation of retrieval + generation02:21
- Quiz – Section 5: Evaluating RAG Systems 5 questions
Section 6: Production Monitoring & Continuous Evaluation3 lectures • 7min
- Designing evaluation in production environments02:05
- Integrating eval into CI/CD or workflow pipelines02:21
- Alerting, thresholds, and incident response02:27
- Quiz – Section 6: Production Monitoring & Continuous Evaluation 5 questions
Section 7: Human Review & Cost Optimization3 lectures • 7min
- Creating scalable human-in-the-loop review systems02:14
- Balancing eval quality vs budget constraints02:07
- Token and model selection strategies to reduce costs02:17
- Quiz – Section 7: Human Review & Cost Optimization 5 questions
Course Conclusion – Key Takeaways1 lecture • 6min
- Course Conclusion – Key Takeaways05:41
From learning to practice
Try this next.
Create a small evaluation set and record both successful responses and observed errors.
A useful next step
Choose where to start.
Talk through your goals in a free 30-minute discovery call with Ted. Bring a task you want to improve and the tools you already use.
Book a discovery callExplore DoJo membership