Stage 05 · measure
Evals
Two evals run through the app's real entry points on hand-labelled cases, graded by code wherever code can grade, and by a judge model only for writing quality. CI replays the recorded run on every pull request, so a prompt or code change that breaks a stage fails the build. Cases live in samples/evals/.
Linking · 53 cases
Prompt link-v1 · claude-opus-5-5 · embeddings voyage-3.5-lite-512 · $0.168 per live run
- Correct decision
- 98% ±4 pts · n=53
- Retrieval recall@5
- 96% ±6 pts · n=46 · right entry retrieved
- Settled by rules
- 11% ±9 pts · n=53 · no model call
| Difficulty | Correct | Cases |
|---|---|---|
| abbreviated | 100% | 5 |
| accents | 100% | 3 |
| ambiguous | 100% | 3 |
| destination | 100% | 6 |
| exact | 100% | 4 |
| meta | 100% | 2 |
| near-miss | 100% | 6 |
| no-match | 100% | 4 |
| semantic | 95% | 20 |
- demo-author-6 “Author: Northbound Staff”: answered new, expected link:author-desk. Cause: retrieval never surfaced the right entry.
Agent · 19 posts
Prompt agent-v1 · 3.8 turns on average (max 5) · agent $0.379 + judge $0.007 (claude-haiku-5-5) per live run
- Submitted
- 100% ±0 pts · n=19
- Field rules
- 100% ±0 pts · n=19 · code
- Grounded
- 100% ±0 pts · n=19 · code
- Region
- 100% ±0 pts · n=19 · vs gold
- Judge: faithful
- 100% ±0 pts · n=14 · Haiku
- Judge: specific
- 100% ±0 pts · n=14 · Haiku
- Judge: plain
- 100% ±0 pts · n=14 · Haiku
- Judge: purpose
- 100% ±0 pts · n=14 · Haiku
Every metric is at or near its ceiling. On these cases the evals guard against regressions; showing that a new prompt is better would need harder cases.
Is the judge trustworthy?
- Known-bad inputs (empty, “I don't know”, the wrong article, hype): every one failed, as it should.
- Not calibrated yet: a person labels the texts blind at /evals/label (locally), then runs npm run eval -- calibrate.