Transit Fare Policy Assistant — evaluation
A narrow retrieval assistant for California transit fare policy, wrapped in a public evaluation harness. The harness is the point; the assistant exists so the harness has something to grade.
The evaluation report
The generated scoreboard, representative failures with full traces, and the safety case.
Improvement over time
Per-suite pass rates across recorded runs, grouped by instrument (a mock and a live run measure different things and are never plotted together).
How the harness is validated
- Self-test and mutation testing — proof the gate catches planted defects
- Coverage map — tested vs untested fare provisions
- Score robustness — confidence intervals and jackknife