Treat Evaluations as a Product Contract
Evals are not a research artifact. They are the contract between your model and the team operating it.
Most teams treat evaluations as research output. The teams that ship reliably treat them as a contract.
A contract names what the system must do, what it must refuse, and how performance is measured. Without one, every release is a renegotiation.
Three properties of a useful eval
It must be reproducible by anyone on the team. It must reflect the production distribution. It must fail loudly when the model regresses.
Most internal evals miss at least one of these. The result is a slow drift between what leadership thinks the model does and what it actually does.
Owning the contract
Pair every eval with an owner and a thresholds doc. When a metric moves, the owner gets paged — not the broader research team.
