A4 — Verifiable Agent Task
RL · Environments · Verification
Hard
Prerequisites (this site)
Readings
Problem
Construct one automatically verifiable coding-agent task from an open-source repository.
Input
Specify (problem, environment, verification system); include a hidden verifier.
Output
Environment spec + hidden verifier + failure taxonomy + anti-reward-hacking checklist.
Requirements
- Reproducible from a clean checkout.
- Reports a baseline and a metric with a confidence interval where applicable.
- Documents ≥2 expected failure modes and reproduces one.
Constraints
- No reference-implementation copying for the mechanism under test.
- Report where your method loses, not only where it wins.
Hints
- Fix all variables except the one you are sweeping.
- Instrument cost (FLOPs/bytes/time) before tuning quality.
- Write the honest limitation before writing the conclusion.
Assignment for CS/AI 684.