Reinforcement learning foundations (optional)

Why it matters

RL is optional for most AI roles, but it is the language of RLHF and of reinforcement-based post-training for reasoning, and it is essential for robotics and agent research. Interviewers in those areas may ask you to derive a policy gradient or explain why PPO clips; elsewhere you mainly need to explain how preference-based fine-tuning relates to RL.

Core ideas

An MDP has states, actions, transition probabilities, rewards and a discount γ. The value V^π(s) is the expected discounted return from s under policy π; Q^π(s, a) also fixes the first action. Bellman equations write each as immediate reward plus discounted next-state value, which yields value and policy iteration when the model is known and temporal-difference learning when it is not.

Policy gradients optimize a parameterized policy directly. Weighting ∇log π(a|s) by the return gives an unbiased gradient estimate (the log-derivative trick). Subtracting a state-dependent baseline, typically a learned value function, keeps it unbiased and reduces variance. Using an advantage estimate A(s, a) = Q(s, a) − V(s) gives actor-critic methods, and generalized advantage estimation trades bias for variance through a parameter λ.

Vanilla policy gradient is sensitive to step size: one large update can collapse the policy. PPO maximizes a surrogate built on the probability ratio between the new and old policy, clipped so that pushing the ratio outside a small interval earns nothing extra, and it reuses each batch of experience for several epochs of minibatch updates.

RLHF for language models treats a response as a sequence of actions. A reward model is trained on human preference pairs, commonly with a Bradley-Terry likelihood, and the policy is optimized with PPO to maximize reward minus a KL penalty to the supervised model, which limits drift and reward exploitation. DPO shows that for this KL-regularized objective the optimal policy can be fitted directly from preference pairs with a classification-style loss.

Common misconceptions

• "A baseline biases the gradient." Any baseline that does not depend on the action leaves the expected gradient unchanged; it only reduces variance.
• "PPO's clip is a hard trust region." Clipping removes the incentive to move further, but the policy can still drift beyond the range over several epochs; implementations monitor KL or stop early.
• "A higher reward-model score means a better model." Optimizing a learned reward too hard exploits its errors; true quality can fall while the proxy keeps rising.
• "One good RL run shows the method works." RL results vary strongly across seeds and implementation details; report several seeds and the details.

How to approach it

Read Spinning Up parts 1 to 3, implement REINFORCE with a baseline on a small control task, then read the PPO page and paper and add clipping. Connect to RLHF through the methods section of the InstructGPT paper and the DPO paper guide. Report every RL result over at least three to five seeds.