Unit 10 · AI research · Lesson 9 of 9
Robot learning foundations (optional)
Step 1: Learn the ideas
- Behavior cloning as supervised learning
- Covariate shift and compounding errors
- DAgger and interactive imitation
- Multimodal action distributions, action chunking and diffusion policies
- RL for control and reward design (awareness)
- Reality gap: visual and dynamics randomization
- System identification and online adaptation
- Evaluating policies on real hardware
Why it matters
Robot learning applies the ideas of this branch under physical constraints: data is expensive, evaluation needs real trials, and simulation is cheap but systematically wrong. If you are not targeting robotics or embodied AI, awareness is enough; if you are, interviewers expect you to discuss imitation-learning failure modes and how you would evaluate a policy honestly.
Core ideas
Behavior cloning fits a policy to expert state-action pairs with supervised learning. Its weakness is covariate shift: the learner's small mistakes take it to states the expert never visited, where its errors are larger, so errors compound with the horizon. DAgger fixes the data rather than the model: run the learned policy, ask the expert to label the states it actually reaches, aggregate and retrain, which turns imitation into a no-regret online learning problem.
Demonstrations are often multimodal, for example passing an obstacle on the left or on the right. A unimodal Gaussian policy averages the modes into an invalid action. Recent methods predict short sequences of actions or sample actions with a diffusion model, which represents several modes and reduces compounding between steps.
Sim-to-real. Simulators differ from reality in appearance, contact dynamics, latency and sensing. Visual domain randomization trains on widely varied textures, lighting and cameras so the real world looks like one more variation; dynamics randomization varies masses, friction and delays. System identification instead measures real parameters to make the simulator closer, and adaptation methods infer them online.
Evaluation. Real-robot results usually rest on tens of trials per condition, so success rates carry wide confidence intervals. Fix the number of trials, the initial-state distribution and the reset procedure in advance, report intervals and failure examples, and include simple baselines.
Common misconceptions
- "Behavior cloning fails because the network is too small." The main failure is distribution shift from compounding errors, which more capacity does not fix.
- "More randomization is always better." Very wide ranges can make the task harder to learn and hurt performance; ranges need justification.
- "A high success rate in a demo shows reliability." Without trial counts, a reset protocol and failure cases, the number is not interpretable.
How to approach it
Use the CS 185/285 imitation-learning lectures and homework to implement behavior cloning and DAgger in simulation, read the DAgger and randomization papers, then skim Diffusion Policy for modern practice. Treat evaluation design as part of the answer to any robot-learning question.
Suggested order
- 1Study the CS 185/285 behavioral-cloning lectures and work through Homework 1 (Imitation Learning).
- 2Read the DAgger paper’s introduction and algorithm section, focusing on why errors compound and how aggregation fixes it.
- 3Read the visual domain-randomization paper and the dynamics-randomization paper (introductions and method sections).
- 4Skim Diffusion Policy for multimodal actions and action-sequence prediction.
- 5Optional: browse LeRobot to see how public robot datasets and imitation-learning policies are packaged.
Step 2: Check off what you can do
Tap each one you can genuinely do or explain. Each earns 10 XP.
Step 3: Go deeper
Berkeley CS 185/285: Deep Reinforcement LearningStart here
UC BerkeleySection: Homework 1: Imitation Learning (handout)
Hands-on behavior cloning and DAgger with lectures that explain distribution shift.
Open(opens in a new tab)Link responded, 5 Oct 2026A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningStart here
arXivFoundational analysis of compounding errors and the DAgger algorithm.
Open(opens in a new tab)Link responded, 5 Oct 2026Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
arXivVisual domain randomization for transferring perception from simulation to a real robot.
Open(opens in a new tab)Link responded, 5 Oct 2026Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
arXivRandomizing physics parameters so control policies survive the reality gap.
Open(opens in a new tab)Link responded, 5 Oct 2026Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
arXivModern policy class for multimodal demonstrations and action sequences.
Open(opens in a new tab)Link responded, 5 Oct 2026
Step 4: Practice
Implement behavior cloning first and measure how performance degrades with horizon; then add DAgger and compare with the same number of expert labels. Report success rates with confidence intervals over a fixed set of evaluation episodes.
- Berkeley CS 185/285 homeworks: imitation learning and policy gradientsOpen(opens in a new tab)Not attemptedUC Berkeley CS 185/285
Record attempts on each problem’s page (+15 XP, +10 more when solved on your own).
Step 5: Test yourself
Answer each one out loud or on paper before you finish the lesson.
- Question 1Why does behavior cloning degrade with longer horizons?
- Question 2What does DAgger require from the expert that behavior cloning does not?
- Question 3Why can a Gaussian policy fail on multimodal demonstrations?
- Question 4When would you prefer system identification over domain randomization?
- Question 5How many trials do you need before a 10-point success-rate difference is convincing?
You’re done when you can
- Explain compounding errors in behavior cloning and how DAgger addresses them in under two minutes.
- Train behavior cloning and DAgger in simulation and compare them with intervals over evaluation episodes.
- Propose a randomization or system-identification plan for a specific sim-to-real gap.