Evaluation loops that don’t slow you down

We like evaluation that fits inside the sprint, not outside of it.

1. Curate 20 edge cases. They should be ugly: long text, noisy inputs, and tricky exceptions. We keep them in version control so they travel with the code.
2. Write tiny assertions. We store expectations alongside the sample so we can see what broke. For LLM outputs, this could be as simple as a regex match or a minimal score threshold.
3. Automate the run. A single script runs the cases locally and in CI. Here’s a sketch:

Copy
import { evaluate } from './lib/evaluator'
import { samples } from './fixtures/samples'
 
async function main() {
  const results = await Promise.all(samples.map((sample) => evaluate(sample)))
  const failures = results.filter((r) => !r.passed)
  if (failures.