Week 11 — Training Agents on Environments Rather Than Text

RL · Agents · Environments

Advanced

IV — Agent post-training

Overview

Post-training is deliberately conventional — SFT, then RL, then on-policy distillation — and the substantive gains come from task/data/environment construction, not new RL algorithms. A synthesized task is (problem, environment, verification system).

What You Will Learn

Define a task as (problem, environment, verification).
Explain why environments, not algorithms, drive gains.
Build an automatically verifiable coding-agent task.
Design an anti-reward-hacking checklist.

Core Concepts

Task as a triple

(\text{problem},\ \text{environment},\ \text{verification system})

Verifier soundness

The verifier, not the reward, defines success; a hackable verifier teaches the wrong policy.

Prerequisites

Work these pages on this site before the lecture.

Lecture notes

Post-training algorithms in this course are conventional (SFT → RL → on-policy distillation). The work is task construction:

\text{task}=(\text{problem},\ \text{environment},\ \text{verification})

A hackable verifier teaches the wrong policy. Hidden tests, no leaked oracle, and an anti-reward-hacking checklist are part of the assignment, not extras. SWE-bench is the public existence proof that “the unit test is the reward.”

Required readings

Assignment A4
Policy gradient · GAE

Lab / Implementation

Construct one automatically verifiable coding-agent task from an open-source repo: environment spec, hidden verifier, failure taxonomy, anti-reward-hacking checklist (Assignment 4).

Not on this site (paper reading required)

SFT→RL→on-policy distillation pipeline
environment construction
verifier design

Mastery Check

DERIVE: the task triple and verifier soundness
IMPLEMENT: a verifiable coding-agent task
BENCHMARK: pass rate by failure type
DEBUG: a verifier that accepts a shortcut
EXPLAIN: why environments dominate algorithm choice

Part of CS/AI 684 — Efficient Long-Context Agent Systems.