What I Learned from XCS224R: From Q-Learning to Modern Reinforcement Learning

I came into XCS224R comfortable with the basic reinforcement learning setup: states, actions, rewards, value functions, and Q-learning. Once the course reached policy gradients, actor-critic methods, and offline RL, the Bellman equation stopped feeling like the center of everything. I had to think more carefully about how the pieces fit together.

These notes follow that path. They start with value functions, then work through policy optimization, offline RL, goal-conditioned RL, and finally DREAM.

1. What is reinforcement learning learning?

An RL problem starts with a repeated interaction:

s_t \rightarrow a_t \rightarrow r_t, s_{t+1}.

The agent observes state $s_t$ , chooses action $a_t$ , receives reward $r_t$ , and transitions to $s_{t+1}$ . The quantity we usually optimize is not the immediate reward but the return

G_t = r_t + \gamma r_{t+1} + \gamma^2r_{t+2}+\cdots

so a good policy has to produce good trajectories, not just isolated actions.

Two value functions show up throughout the course:

V^\pi(s) = \mathbb{E}_\pi[G_t\mid s_t=s]

and

Q^\pi(s,a) = \mathbb{E}_\pi[G_t\mid s_t=s,a_t=a].

I remember them as two questions:

$V(s)$ : How good is it to be here?
$Q(s,a)$ : How good is it to be here and take this action?

I kept coming back to this distinction, especially when the course reached advantage functions and offline RL.

2. Q-learning and bootstrapping

Q-learning was the first algorithm that made policy improvement feel concrete to me. Its Bellman target is

The Bellman target is

y = r + \gamma(1-d)\max_{a'}Q_{\text{target}}(s',a'),

where $d$ indicates whether the transition terminates the episode.

We train the critic by minimizing something like

\mathcal{L}_Q = \left(Q(s,a)-y\right)^2.

The target contains another prediction. We do not know the true long-term value of $s'$ , so we estimate today’s value using an estimate of tomorrow’s value. That is bootstrapping.

It also explains the target network. If one rapidly changing network produces both the prediction and the target, the optimization has to chase a moving value. A slowly updated target network makes that value more stable.

There is still an assumption hiding in the update: we need a practical way to compute

\max_a Q(s,a).

This is easy for a small discrete action space. It is awkward when actions are continuous, which is one reason to learn a policy directly.

3. Learning the policy directly

A policy

\pi_\theta(a\mid s)

directly describes which actions to take. Policy-gradient methods optimize the policy parameters instead of deriving behavior through an explicit maximization of $Q$ .

The basic policy-gradient idea is

\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} \left[ \nabla_\theta\log P_\theta(\tau)R(\tau) \right].

Because the environment dynamics do not depend on the policy parameters, this eventually turns into terms involving

\nabla_\theta\log\pi_\theta(a_t\mid s_t).

The useful intuition for me is:

Actions associated with unexpectedly good outcomes should become more likely; actions associated with bad outcomes should become less likely.

The raw trajectory return is a noisy learning signal, though. A critic can turn it into a more useful local comparison for the actor.

4. Advantage: better than expected?

The advantage function is

A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s).

The equation is simple, but the comparison took me a while to internalize. Suppose an action has $Q(s,a)=100$ . Is that good? If $V(s)=20$ , yes. If $V(s)=120$ , the action is worse than what the policy normally expects from that state.

Advantage therefore does not ask

“Was the outcome good?”

It asks

“Was this action better than what I should normally expect here?”

With that substitution, the policy update is roughly

\nabla_\theta J \approx \mathbb{E} \left[ \nabla_\theta\log\pi_\theta(a\mid s) A(s,a) \right].

Positive advantage makes the action more likely, while negative advantage makes it less likely. This was the point where policy gradients, value functions, and actor-critic methods clicked together for me.

5. What changes in offline RL

Offline RL was one of the bigger conceptual jumps in the course. In online RL, the agent can interact with the environment while learning. If the critic gives some strange action an enormous value, the policy may try it and collect evidence that the estimate was wrong.

Offline RL has no such correction loop. The agent receives a fixed dataset

We instead receive a fixed dataset

\mathcal D = \{(s,a,r,s',d)\}

and must learn from those transitions alone. Suppose the dataset mostly contains actions near

a\approx0.2,

but the learned actor proposes

a=0.95.

The critic may give this unfamiliar action a huge value because it has little evidence there. The actor then finds the error and exploits it. This is the distribution-shift problem in offline RL: policy optimization searches for weaknesses in the critic, including regions outside the dataset.

6. IQL avoids maximizing an unreliable critic

Implicit Q-Learning pulled several earlier concepts together for me. Its Q update can use

A standard Q update can use

y = r+\gamma(1-d)V(s').

The important detail is what is missing. We do not explicitly compute

\max_{a'}Q(s',a').

Instead, IQL learns a value function that tracks the upper part of the Q-value distribution over actions represented in the dataset. It does this with expectile regression. Given

\delta = Q(s,a)-V(s),

the expectile loss gives different weights to positive and negative residuals. With a high expectile parameter, the loss penalizes underestimates of high-Q actions more strongly. $V(s)$ moves toward the better actions in the dataset without maximizing over actions that may be out of distribution.

Advantage is still

A(s,a)=Q(s,a)-V(s).

The policy is trained using advantage-weighted behavioral cloning, roughly

\mathcal L_\pi = -\mathbb{E}_{(s,a)\sim\mathcal D} \left[ \exp(\beta A(s,a)) \log\pi_\theta(a\mid s) \right].

My shorthand for IQL is behavioral cloning with a preference: imitate the good dataset actions more strongly than the bad ones. The actor improves using actions that appear in the data instead of searching arbitrary actions for critic errors.

7. CQL changes the critic instead

Conservative Q-Learning works on the same extrapolation problem from the critic side. Ordinary Q-learning can assign unrealistically high values to actions outside the dataset, so CQL pushes the critic to be conservative about unsupported actions. Conceptually, we do not want

Q(s,a_{\text{OOD}})

to look attractive just because the critic has little evidence about it.

The comparison I use is:

IQL avoids querying the critic with arbitrary maximizing actions.
CQL trains the critic to give unsupported actions conservative values.

Both are responses to the same offline-RL question:

How can a policy improve without exploiting errors in regions where our dataset provides little evidence?

8. Adding a goal changes the value function

Goal-conditioned RL adds the desired outcome to the policy. Instead of learning

\pi(a\mid s),

we learn

\pi(a\mid s,g),

where $g$ describes what we want to accomplish. Likewise,

Q(s,a,g)

measures an action relative to that goal. One policy can now produce different behavior for different requested goals.

Sparse rewards make this difficult. If the agent almost never reaches the requested goal, nearly every trajectory looks like a failure.

9. HER gives failures another goal

Hindsight Experience Replay was one of my favorite ideas in the course. Suppose the requested goal was $g$ , but the trajectory reached $g'$ . It failed relative to $g$ , yet the same sequence succeeded relative to $g'$ .

HER relabels the experience with goals the agent reached. Instead of discarding a trajectory because

“I failed to reach the requested goal,”

we can reinterpret it as

“I successfully reached a different goal, so what can I learn from that?”

The same transition can now provide a learning signal in a sparse-reward environment. HER gets more use out of the collected experience without pretending that the original goal was reached.

10. DREAM learns what to explore

The last topic I explored was DREAM, which moves from goal-conditioned RL into meta-reinforcement learning. In meta-RL, an agent trains across a family of related problems and then has to adapt quickly to a new one. That adaptation often has two phases:

Exploration gathers information about the new environment.
Execution uses that information to complete the requested task.

Training both phases end to end creates a chicken-and-egg problem. Exploration receives a useful signal only when the execution policy already knows how to use the information it finds. The execution policy, meanwhile, can learn only if exploration has found useful information. Early in training, neither policy is competent enough to teach the other.

DREAM separates their objectives.

During meta-training, the execution policy is given a problem identifier, denoted by $\mu$ . This identifier contains enough information to characterize the environment, but it may also include irrelevant details. An information bottleneck compresses it into a latent representation $z$ intended to retain only what execution needs.

The exploration policy then produces a trajectory $\tau$ that reveals this task-relevant information. Roughly, it tries to make the mutual information

I(z;\tau)

large. After observing the exploration trajectory, the agent should be able to recover the useful context encoded by $z$ .

The kitchen example made the idea concrete for me. An execution policy may need to know where ingredients are stored in a new kitchen, so the exploration policy should open the relevant cupboards and refrigerators. Learning the wall color is useless, even if that observation is novel.

This changed my view of exploration. It does not have to mean random action, broad state coverage, or novelty for its own sake. It can mean gathering the specific information that a later policy will need.

HER asks how to learn more from the goals an agent happened to achieve. DREAM asks a complementary question:

What information should the agent deliberately seek before it tries to execute a task?

For DREAM, the learning problem includes deciding what is worth knowing before the agent acts.

11. How the pieces fit together

XCS224R became easier to follow once I stopped treating each algorithm as a new collection of equations. Most of them begin with a reasonable idea and then address the place where that idea breaks.

Q-learning estimates the long-term value of actions. Policy gradients learn the actions directly, and actor-critic methods use a value estimate to tell the policy whether an action was better than expected.

Offline RL removes the agent’s ability to test and correct a bad estimate. IQL responds by avoiding arbitrary maximizing actions, while CQL makes the critic pessimistic about unsupported ones.

Goal-conditioned RL makes value depend on the requested goal. HER recovers learning signals from goals that were reached instead. DREAM goes one step earlier and asks what information the agent should collect before execution begins.

I no longer start a new RL paper by asking, “What are its equations?” I first ask, “What failure mode is this algorithm trying to fix?” Once I know that, the equations have somewhere to attach.