Porter

WordPressContentful

Stage 05 · measure

Evals

Two evals run through the app's real entry points on hand-labelled cases, graded by code wherever code can grade, and by a judge model only for writing quality. CI replays the recorded run on every pull request, so a prompt or code change that breaks a stage fails the build. Cases live in samples/evals/.

Prompt link-v1 · claude-opus-5-5 · embeddings voyage-3.5-lite-512 · $0.168 per live run

Correct decision
98%
±4 pts · n=53
Retrieval recall@5
96%
±6 pts · n=46 · right entry retrieved
Settled by rules
11%
±9 pts · n=53 · no model call
DifficultyCorrectCases
abbreviated100%5
accents100%3
ambiguous100%3
destination100%6
exact100%4
meta100%2
near-miss100%6
no-match100%4
semantic95%20
  • demo-author-6 “Author: Northbound Staff”: answered new, expected link:author-desk. Cause: retrieval never surfaced the right entry.

Agent · 19 posts

Prompt agent-v1 · 3.8 turns on average (max 5) · agent $0.379 + judge $0.007 (claude-haiku-5-5) per live run

Submitted
100%
±0 pts · n=19
Field rules
100%
±0 pts · n=19 · code
Grounded
100%
±0 pts · n=19 · code
Region
100%
±0 pts · n=19 · vs gold
Judge: faithful
100%
±0 pts · n=14 · Haiku
Judge: specific
100%
±0 pts · n=14 · Haiku
Judge: plain
100%
±0 pts · n=14 · Haiku
Judge: purpose
100%
±0 pts · n=14 · Haiku

Every metric is at or near its ceiling. On these cases the evals guard against regressions; showing that a new prompt is better would need harder cases.

Is the judge trustworthy?

  • Known-bad inputs (empty, “I don't know”, the wrong article, hype): every one failed, as it should.
  • Not calibrated yet: a person labels the texts blind at /evals/label (locally), then runs npm run eval -- calibrate.