KAK Model Evaluation Report — 100-Question Evaluation Set
Evaluation date: 18 October 2026 (MYT) · Set: 100 canonical Bahasa Indonesia questions · Status: provisional, not client acceptance.
See the 100-question evaluation set for per-question answers and results.
This manually selected 100-question evaluation set provides an initial measured view, not evidence of production readiness. It passed the retrieval-recall check and failed the other five configured thresholds. The independent faithfulness and hallucination rates remain null because six claim rulings and five refusal rulings were unavailable.
Scoring rule change, 17 October 2026. The answer, rubric and outcome numbers in this report are scored under figures_only_true; the same runs scored under the earlier 0.80-confidence bar are shown beside them, labeled, throughout. The rule, why it exists, and what it cannot show are in Scoring rule.
Run and provenance
| Item | Value |
|---|---|
| Run id | golden-seed-easy-20261017-172913-170652 (arm easy, judged) |
| Date | 17 October 2026, 17:29–18:03 UTC |
| Build under test | e8a34442 — the commit production runs after the #952 deploy. The isolated eval API reported deployed_ref e8a34442f111871b7ed0078e4f7bb5738994671b, reproducible: true. |
| Index | ragaceh_eval on the box-local eval Postgres (aceh-eval-db), reused with --reuse-index; corpus manifest 2026-10-10.1; 100 scored cases |
| Answering model | qwen3.8-27b (reported by the API), embeddings bge-m3, reranker bge-reranker-v2-m3, OCR model PaddleOCR-VL-0.9B; served context window 32,768 tokens; faithfulness and Policy Guard enabled; external LLM disabled |
| Judge | Jev jev-1.13.0 through typesafe-sdk 0.7.2, confidence threshold 0.8. The judge model, questions, equivalences, rubric and mapping hashes are identical to the 17 October run's; adapter_sha256 differs because that hash covers the adapter source, which now carries the figures_only_true rule this build adds. Both records are frozen and committed. |
| Records | eval/golden-seed/easy-runs/golden-seed-easy-20261017-172913-170652-easy.json, its -faithfulness.json and -judge-config.json |
Results
All values below come from a 100-case evaluation on 18 October 2026. The six threshold checks are listed separately below.
| Metric | Result | Denominator / interpretation |
|---|---|---|
| Recall@5 | 89.29% | 75/84 expected-document cases |
| MRR@5 | 0.875 | Mean reciprocal rank across 84 cases |
| nDCG@5 | 0.8797 | Mean normalized discounted gain across 84 cases |
| Answer-rubric pass | 72.00% | 72/100; judged answer/refusal rubric passed |
| Answer-correct rate | 72.00% | 72/100: 63 correct answers, 8 correctly handled expected refusals and 1 rubric-passing answer whose expected route contract failed |
| Outcome pass | 71.00% | 71/100 after expected outcomes and any route contract are applied |
| Citation presence | 86.36% | 76/88 expected-answer cases had at least one citation; presence is not support |
| Refusal pass | 66.67% | 8/12 expected-refusal cases passed |
| Route pass | 66.67% | 2/3 route checks passed |
| Total latency, p50 / p95 / p99 | 13.7 / 61.6 / 74.4 s | All 100 cases; search plus query |
| Search latency, p50 / p95 / p99 | 0.377 / 0.589 / 0.721 s | All 100 cases |
Configured thresholds: 1/6 pass, 5/6 fail. Recall@5 ≥80% passed. Route pass =100%, answer-rubric pass ≥80%, outcome pass ≥80%, citation presence =100% and refusal pass =100% failed. Overall, the evaluation did not pass.
This is the first run scored with figures_only_true applied live rather than by a re-score of a stored run, so it carries no second set of numbers: re-scoring the committed record offline changes 0 cases and leaves every metric identical.
Scoring rule: figures_only_true (17 October 2026)
The judge performs separate checks for each case and marks each of its own answers with a confidence, |2 × p_yes − 1|. Any check below the 0.8 bar was "flagged", and one flagged check made the whole case unavailable — the verdict withheld and the case kept in the answer and rubric denominators without contributing a pass. That treated a hedge on the figures check, a question about whether the answer's numbers are right, as if the answer had failed.
The rule now: when the figures check is the only flagged check on a case and the judge's own answer to that check is yes (p_yes at or above the spec's 0.5 threshold), the flag is cleared and the case is graded from its stored verdict. Nothing else moves. A figures hedge that leans no (p_yes below 0.5) keeps its flag — an unsure "the numbers are wrong" is not a pass. A figures flag beside any other flag (support, coverage, recall) keeps its flag; those are doubts about a named claim, not about a question the answer had nothing to say about. Refusal cases never ask the figures question and are unaffected. The rule lives in one place, src/aceh_rag/eval/jev_judge.py judge_flags(), which the live run, the re-scorer and this report all read.
Why: judges hedge the figures question on answers that state no numbers at all, which its own instructions already allow ("if the answer makes no numeric claims and none are required, answer yes"). Across the 200 case-runs in the calibration sample, 56 carried a flagged figures check as their only flag; the rule clears the 37 where the judge's own answer was yes — 28 correct answers, one correct-with-a-blemish, and 8 wrong refusals that move from unsure to a decisive fail — and leaves the 19 that lean no flagged. On the 17 October run it moved 16 cases from unavailable to correct and 4 from unavailable to a decisive wrong refusal that had been reported as unsure. The 18 October run is the first scored with the rule already in force, so it has no second set of numbers; re-scoring its stored checks offline changes no case.
The limit of the evidence. The rule is supported by a sample in which only 3 of the 60 wrong case-runs are assertive wrong answers; the other 57 are wrong refusals. "Zero wrong answers passed" is therefore 0/60 on a sample that cannot price the case this rule is meant to help with — a fluent, confident wrong number that the figures check endorses. The rule is unproven there. The 56 ground-truth labels are also a single rater's, and the hedged probabilities were read while that ground truth was being established, so the split between tuning and validation sets is a pre-registered division, not a blind trial.
How these numbers were produced. The published run is a live judged run of the current build: the answers, retrieval results and judge rulings in eval/golden-seed/easy-runs/golden-seed-easy-20261017-172913-170652-easy.json are its own, and the rule was applied by the runner as it graded. Re-scoring that record offline from its stored checks (scripts/rescore_golden_seed.py --from-stored-checks) changes nothing; the step is run only because the table generator requires the rule attribution it writes. The earlier runs' figures under the rule are offline re-scores of their stored checks, which recompute the flag set, the answer axes and the verdicts and nothing else. The per-question tables were regenerated from the 18 October run with scripts/generate_docs_evaluation_table.py; the exact commands are in eval/golden-seed/easy-runs/README.md.
Metric definitions
| Metric | Plain-language meaning and formula |
|---|---|
| Recall@5 | For each case with expected source documents, relevant documents in the first five results ÷ expected relevant documents; averaged over retrieval cases. |
| MRR@5 | For each retrieval case, 1 / rank of the first expected document in the top five, or zero if none appears; averaged over cases. |
| nDCG@5 | Binary relevance gain at each rank, discounted by 1/log2(rank + 1), divided by the ideal top-five gain; averaged over cases. |
| Answer-rubric pass | Cases that pass the answer-and-figure rubric, or the expected-refusal rubric, ÷ all 100 cases. A flagged judge check fails closed, except the one figures case above. |
| Answer-correct rate | Cases whose answer is correct ÷ 100. For judged answered cases, correctness requires the judged answer/figure rubric and citation support; derive_answer_axes() does not require source_span_found. Expected refusals are correct only when the refusal is judged and its citation/source conditions pass. The report's own faithfulness_pass_rate is this number relabelled (ADR-0042), not the independent pass below. |
| Outcome pass | Cases with outcome_pass=true ÷ 100; starts from answer correctness and also requires any expected route contract to pass. |
| Citation presence | Expected-answer cases with at least one citation ÷ 88 expected-answer cases. It does not say whether the citation supports the answer. |
| Refusal pass | Passing expected-refusal cases ÷ 12 expected-refusal cases. An unnecessary refusal on an expected-answer case is a failure, not a refusal pass. |
| Latency percentile | Sorted case latencies, linearly interpolated at index (n - 1) × quantile; total latency is search plus query. |
| Faithfulness claim-support rate | Supported factual claims ÷ factual claims judged. contradicts and says_nothing are unsupported; unavailable rulings stay in the denominator and are not support. |
| Faithfulness answer-pass rate | Answered outputs with no unsupported claim ÷ answered outputs with at least one factual claim judged. Refusals are not in this denominator. |
| Hallucination rate | Answered outputs with at least one unsupported claim ÷ all answered (non-refused) outputs. Unavailable rulings make this rate null, not zero. |
Retrieval formulas are implemented in src/aceh_rag/eval/evaluation.py:238-269; answer/refusal aggregation is in src/aceh_rag/eval/golden_seed.py:2004-2086,6173-6205; latency interpolation is in src/aceh_rag/eval/report_breakdowns.py:75-82. Faithfulness definitions and fail-closed handling are in src/aceh_rag/eval/faithfulness.py:101-140,209-367 and eval/README.md#metrics.
What happened in the 29 non-correct cases
The 29 cases outside the 71 correct outcomes were 24 judge-unsure, 4 decisively wrong and 1 partial. No confidently wrong case published a wrong number: all four decisive failures are coverage_incomplete refusals on questions expected to be answered, each with no citation. Among the 24 judge-unsure cases, 20 expected an answer and four expected a refusal; all four refusals turned on a refusal ruling that fell below the 0.8 confidence threshold. The run had no HTTP errors, timeouts, unscoreable answers, or empty-answer cases.
The 29 non-correct cases by what the answer path published: 12 refusals (10 coverage_incomplete, 1 over_refusal, and 1 completion the output boundary could not parse) and 17 answers, of which 11 carry a flagged judge check and one is published with the no_claims marker. Fourteen of the 88 expected-answer outputs were refusals; the other 74 published an answer.
Expected-span evidence gives a useful retrieval split: of the 29 non-correct cases, 23 had an applicable expected source span; 18 of those spans appeared in top-five /search results and five did not. Six cases had no applicable passage span. Separately, 12 of the 88 expected-answer outputs had no citation. The harness checks source_span_found against top-k search passages before synthesis; it does not prove the answer used that passage (src/aceh_rag/eval/golden_seed.py:1807-1815).
| Example | Outcome | Evidence |
|---|---|---|
b2-seed-015 — Qanun 7/2019 written-notice deadline | Partial | The answer is correct, grounded in two cited passages and passes the rubric, but the response recorded no search_documents call, so the expected route contract failed. |
b2-de-j25-01 — uqubat range for pelecehan seksual after Qanun 12/2025 | Judge unsure | The answer path refused with coverage_incomplete although the expected span was in the top-five results; the judge flagged the support check. |
b2-bps-003 — Aceh labour force 2024 | Judge unsure | The live BPS row was refused as an over-refusal; the judge flagged figure equivalence on the refusal. |
b2-de-r-04 — refused-information requests at PPID Aceh 2023 | Judge unsure | An expected refusal; the reply explains that the SatuData dataset is empty and cites seven passages, and the judge's refusal ruling fell below the confidence threshold. |
b2-seed-030 — information requests received in 2022 | Judge unsure | The output boundary rejected the completion, so the case fell back to a canned refusal; the judge flagged the figures and coverage checks. |
“Partial” is a specific judge-grade bucket, not a count of partly correct facts: after unscoreable and flagged cases are excluded, it means an answered case passes the answer rubric but fails full outcome correctness. Refusals are graded against the expected outcome.
Independent faithfulness and hallucination pass
The standalone pass assessed the same 100 recorded answers.
It used a frozen judge configuration with a 0.8 confidence threshold. The results include 100 per-answer records and a deterministic 10-claim human-review sample; that sample has not been reviewed by a person.
| Evidence | Result |
|---|---|
| Factual claim rulings | 173 total: 167 supported, 0 ruled unsupported, 6 unavailable |
| Answered outputs / confidently classified refusals | 79 / 21 |
| Answered outputs with at least one factual claim judged | 79; 71 passed, 8 unavailable |
| Refusal rulings unavailable | 5 |
| Answers with no cited passages | 1 |
| Whole-answer fallback | 1 answer |
| Official claim-support, answer-pass, hallucination rates | null / null / null |
The official rates remain null because the frozen policy fails closed if any claim or refusal ruling is flagged or below the 0.8 confidence threshold. No claim was ruled unsupported, but that does not establish a 0% hallucination rate while six claim rulings and five refusal rulings remain unavailable.
The supplemental resolved-case yield below does not replace the official null rates:
- Resolved-case answer-pass yield: 71/92 = 77.17%. This excludes the eight answered outputs with unavailable claim or refusal rulings; the other 92 comprise 71 fully judged answer passes and 21 refused outputs. Refusals are not counted as passing answers. This is an all-outcome yield, not the defined
faithfulness_answer_pass_rate.
The five cases with an unavailable refusal ruling and their evidence:
| Case | Unavailable ruling and evidence |
|---|---|
b2-seed-052 — expected refusal, two cited passages | The standalone pass marks the refusal ruling unavailable; its one claim was supported and none was ruled unsupported. |
b2-de-sd-03 — expected answer, two cited passages | The standalone pass marks the refusal ruling unavailable; of its two claims one was supported and one was unavailable, and none was ruled unsupported. |
b2-de-r-04 — expected refusal, seven cited passages | The standalone pass marks the refusal ruling unavailable; its one claim was supported and none was ruled unsupported. |
b2-de-r-05 — expected refusal, four cited passages | The standalone pass marks the refusal ruling unavailable; all three claims were supported and none was ruled unsupported. |
b2-pg-jail-id-002 — expected refusal, no cited passages | The standalone pass marks the refusal ruling and its only claim ruling unavailable; no claim was ruled unsupported. |
For these cases, a flagged check or confidence below 0.8 made the ruling unavailable. The independent review reports unavailable flags and counts, but not the exact refusal confidence or whether a check was flagged versus below threshold; that detail cannot be recovered. The cases stay outside the 92-outcome ratio and are not treated as passes. The main report's 72% faithfulness pass figure is the answer-correct rate relabelled, not the independent faithfulness result. The 2/100 draft-claim gate-drop rate is a blocking diagnostic, not a hallucination rate.
Set construction and limits
The 100-question evaluation set was manually selected from the 632-row label bank. It contains 100 canonical Bahasa Indonesia questions: JDIH 48, PPID 42, Policy Guard 6, BPS 3, and OpenData 1. It has 96 static and four live cases. The selection favors less complex questions than the full evaluation set, covers single-turn questions only, and contains no English questions. It is not held out and is not a client-provided golden set. See eval/README.md for selection caveats.
Of the 100 rows, 84 name an expected source document (26 distinct documents); 81 have an expected source span (80 exact-text matches and one harness reflow match). Three spreadsheet rows use gold_cells but have no passage-scoring span; they are excluded from passage-span scoring, while document recall still applies. Sixteen rows name no source document: seven expected refusals, six Policy Guard cases, and three live BPS cases. The harness has no spreadsheet-cell pointer scoring path (eval/README.md#easy-baseline).
The evaluation labels remain draft pending human review, and a manually selected set can overstate performance. Even zero failures on 100 representative independent cases would bound the failure rate only to about 3% at 95% confidence (rule of three); roughly 30,000 zero-failure cases would be needed to bound it near 0.01%. This run had failures and judge-uncertain cases. The UAT target of 99.99% or more is not demonstrated by this evaluation. A possible replacement for client agreement is a frozen, human-reviewed set with a defined sample size and confidence interval, a target such as ≥95% faithfulness, and zero unsupported published claims. That proposal is not yet agreed.
Not measured here: English; multi-turn conversation memory; structure counts, amendments, and multi-hop legal questions; broad district coverage (there is one district OpenData probe); OCR quality; upload, login, concurrency, and citation-link-open behavior. Live sources can drift. The four live cases are not static retrieval tests.
The evaluation index report did not verify a live document listing or checksums. A 100-case sample found no undeclared documents, but that does not prove the index contained only the seed corpus.
Comparison with earlier runs
Runs of this set
| Date (UTC) | Run id | Build under test | Environment | Mode | Recall@5 | MRR@5 | nDCG@5 | Answer-correct | Citation presence | Refusal pass | Route pass |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026-10-15 10:52 | golden-seed-easy-20261015-105206-652591 | 99836b20 | box eval API | search only | 89.29% | 0.8750 | 0.8797 | — | — | — | — |
| 2026-10-15 14:28 | golden-seed-easy-20261015-142858-143088 | 99836b20 | box eval API | judged · 0.80 bar | 89.29% | 0.8750 | 0.8797 | 34.00% | 46.59% | 83.33% | 100.00% |
| ↳ same run, re-scored 17 Oct 2026 | golden-seed-easy-20261015-142858-143088 | 99836b20 | box eval API | judged · figures_only_true | 89.29% | 0.8750 | 0.8797 | 47.00% | 46.59% | 83.33% | 100.00% |
| 2026-10-16 01:10 | golden-seed-easy-20261016-011013-713214 | 99836b20 | box eval API | search only | 89.29% | 0.8750 | 0.8797 | — | — | — | — |
| 2026-10-16 02:19 | golden-seed-easy-20261016-021907-168083 | 0475efa8 (PR #927 head) | box eval API | search only | 89.29% | 0.8750 | 0.8797 | — | — | — | — |
| 2026-10-16 10:01 | golden-seed-format-fix-20261016-100126 | abb7cc7d (PR #934 head) | private lab | judged · 0.80 bar | 89.29% | 0.8750 | 0.8797 | 52.00% | 92.05% | 50.00% | 66.67% |
| ↳ same run, re-scored 17 Oct 2026 | golden-seed-format-fix-20261016-100126 | abb7cc7d (PR #934 head) | private lab | judged · figures_only_true | 89.29% | 0.8750 | 0.8797 | 69.00% | 92.05% | 50.00% | 66.67% |
| 2026-10-17 01:33 | golden-seed-easy-20261017-013316-184015 | a79a7355 | box eval API | judged · 0.80 bar | 89.29% | 0.8750 | 0.8797 | 55.00% | 85.23% | 75.00% | 100.00% |
| ↳ same run, re-scored 17 Oct 2026 | golden-seed-easy-20261017-013316-184015 | a79a7355 | box eval API | judged · figures_only_true | 89.29% | 0.8750 | 0.8797 | 71.00% | 85.23% | 75.00% | 100.00% |
| 2026-10-17 17:29 | golden-seed-easy-20261017-172913-170652 | e8a34442 (post-#952) | box eval API | judged · figures_only_true | 89.29% | 0.8750 | 0.8797 | 72.00% | 86.36% | 66.67% | 66.67% |
A ↳ row is the run above it, scored under the rule in force from 17 October 2026 instead of the 0.80-confidence bar; the run, its answers, its retrieval and its judge rulings are identical in both. The 15 October and private-lab 16 October re-scored rows come from retained records that are not committed to this repository. The 17 October record is committed; its re-scoring command is in eval/golden-seed/easy-runs/README.md. The last row is the published run, judged live on the build that ships the rule.
Retrieval is flat across every run: recall@5 89.29%, MRR@5 0.875 and nDCG@5 0.8797 on the same 84 retrieval cases, including the two runs that carried the earlier retrieval change (PR #927) and the 18 October run that followed it. The answer layer moved instead: under figures_only_true answer-correct goes from 47% (15 October) to 71% (17 October as republished) to 72% (18 October), which is +25.00 pp against the 15 October baseline and +1.00 pp against the latest earlier run. Under the 0.80 bar the same baseline-to-17-October comparison was 34% to 55%. What moved the wrong way is refusal pass (83.33% → 75.00% → 66.67%) and route pass (100% → 100% → 66.67%); both are covered under What happened in the 29 non-correct cases and What limits comparability.
Change across the runs compared here
The 18 October run is judged on e8a34442 (its UTC stamp is 20261017-172913; the run started 17 October 17:29 UTC and finished 17 October 18:03 UTC, which is 18 October in MYT, the calendar this report's dates use); the 17 October run appears twice — as it was scored under the 0.80-confidence bar and as republished under figures_only_true — because the rule change, not the build, moved its answer numbers. The 16 October run (arm fixed100, in a private lab on PR #934's branch head) and the three search-only runs stay in the run list for continuity but are not columns here: the 16 October run is not like-for-like, and the search-only runs measured no answers.
The 15 October baseline, republished 17 October and 18 October columns use figures_only_true. The “17 Oct #939 as scored” column retains the earlier 0.80-confidence bar. Under that earlier bar, the 15 October baseline scored 34.00% / 34.00% / 34.00% / 27.27% for answer-rubric pass, answer-correct, outcome pass and citation correctness. Every other row is unchanged by the rule. The two #939 columns are the same run; they differ only in those four answer-layer rows.
| Metric | 15 Oct baseline | 17 Oct #939 as scored | 17 Oct #939 republished | 18 Oct rerun | Change vs 15 Oct | Change vs 17 Oct republished |
|---|---|---|---|---|---|---|
| Recall@5 | 89.29% | 89.29% | 89.29% | 89.29% | +0.00 pp | +0.00 pp |
| MRR@5 | 0.8750 | 0.8750 | 0.8750 | 0.8750 | +0.00 | +0.00 |
| nDCG@5 | 0.8797 | 0.8797 | 0.8797 | 0.8797 | +0.00 | +0.00 |
| Answer-rubric pass | 47.00% | 55.00% | 71.00% | 72.00% | +25.00 pp | +1.00 pp |
| Answer-correct | 47.00% | 55.00% | 71.00% | 72.00% | +25.00 pp | +1.00 pp |
| Outcome pass | 47.00% | 55.00% | 71.00% | 71.00% | +24.00 pp | +0.00 pp |
| Citation presence | 46.59% | 85.23% | 85.23% | 86.36% | +39.77 pp | +1.14 pp |
| Citation correctness | 42.05% | 52.27% | 70.45% | 72.73% | +30.68 pp | +2.27 pp |
| Refusal pass | 83.33% | 75.00% | 75.00% | 66.67% | -16.67 pp | -8.33 pp |
| Route pass | 100% | 100% | 100% | 66.67% | -33.33 pp | -33.33 pp |
| Answer-bearing recall@5 | 89.66% | 93.10% | 93.10% | 94.83% | +5.17 pp | +1.72 pp |
| Distractor avoidance | 70.00% | 70.00% | 70.00% | 70.00% | +0.00 pp | +0.00 pp |
| Total latency p50 | 13.98 s | 12.76 s | 12.76 s | 13.66 s | -0.32 s | +0.90 s |
| Total latency p95 | 60.78 s | 49.64 s | 49.64 s | 61.56 s | +0.78 s | +11.92 s |
| Total latency p99 | 79.57 s | 54.10 s | 54.10 s | 74.37 s | -5.20 s | +20.28 s |
| Search latency p50 | 0.405 s | 0.405 s | 0.405 s | 0.377 s | -0.028 s | -0.028 s |
| Search latency p95 | 0.620 s | 0.639 s | 0.639 s | 0.589 s | -0.032 s | -0.050 s |
| Search latency p99 | 0.804 s | 0.704 s | 0.704 s | 0.721 s | -0.083 s | +0.017 s |
| Factual claim rulings | 95 | 149 | 149 | 173 | +78 | +24 |
| Claims supported | 93 | 143 | 143 | 167 | +74 | +24 |
| Claims ruled unsupported | 0 | 0 | 0 | 0 | +0 | +0 |
| Claims unavailable | 2 | 6 | 6 | 6 | +4 | +0 |
| Answered outputs / refusals | 43 / 57 | 78 / 22 | 78 / 22 | 79 / 21 | — | — |
| Faithfulness answer passes | 41 | 70 | 70 | 71 | +30 | +1 |
| Refusal rulings unavailable | 2 | 3 | 3 | 5 | +3 | +2 |
| Official claim-support / answer-pass / hallucination rate | null | null | null | null | — | — |
Questions that flipped
Both sides of these comparisons are scored under figures_only_true, read from the two committed run records. The lists come from per-case outcome_pass on both sides: scripts/compare_golden_seed.py refuses this pair, because it requires identical frozen judge configurations and the adapter hash covers the adapter source, which now carries the rule this build adds (see What limits comparability).
Against the 17 October run as republished: 10 fail → pass and 10 pass → fail. Outcome pass is unchanged at 71/100; the movements offset, so this run is not an improvement over the latest earlier run on this metric and its answer-correct edge is one case.
Fail → pass (10): b2-bps-002, b2-de-r-01, b2-de-sd-02, b2-de-wn-03, b2-jdih-mh-0005, b2-ppid-019, b2-seed-004, b2-seed-018, b2-seed-037, b2-sheet-C-1921-sum. Four are published answers whose flagged judge check has cleared (b2-jdih-mh-0005, b2-seed-004, b2-seed-037, b2-sheet-C-1921-sum), one is an expected refusal whose refusal ruling recovered (b2-de-r-01), two were unresolved_citation fallbacks that now publish a cited answer (b2-de-sd-02, b2-de-wn-03), two were coverage_incomplete refusals that now answer (b2-ppid-019, b2-seed-018), and one is the live b2-bps-002, which the BPS tool now answers instead of over-refusing.
Pass → fail (10):
| Question | 17 Oct | 18 Oct | Main cause |
|---|---|---|---|
b2-de-j25-01 | Pass | Fail | Refuses with coverage_incomplete where it previously answered; the judge also flagged the support check. |
b2-de-q7-03 | Pass | Fail | Same class: coverage_incomplete refusal, with figures and support flags. |
b2-de-rd-02 | Pass | Fail | Same class: coverage_incomplete refusal, no judge flag. |
b2-new-nonmuslim | Pass | Fail | Same class: coverage_incomplete refusal, no judge flag. |
b2-seed-043 | Pass | Fail | Same class: coverage_incomplete refusal, no judge flag. |
b2-seed-048 | Pass | Fail | Same class: coverage_incomplete refusal, with a figures flag. |
b2-de-r-04 | Pass | Fail | Expected refusal; the reply now explains that the SatuData dataset is empty and cites seven passages, and the judge's refusal ruling fell below the 0.8 threshold, so it fails closed. |
b2-de-r-05 | Pass | Fail | Expected refusal; the reply was rewritten and the judge's refusal ruling fell below the threshold. |
b2-de-bm21-01 | Pass | Fail | The answer was rewritten at greater length and now carries six citations; the judge flagged the figures and support checks, so the verdict is withheld. |
b2-seed-015 | Pass | Fail | The answer is still correct and cited, but the response recorded no search_documents call, so the declared route contract failed. Reproduced on demand: the agent answers this question in a single call with no tool part, while b2-seed-001 and b2-seed-037 still open with search_documents. |
Against the 15 October baseline: 32 fail → pass and 8 pass → fail. Twenty-seven of the gains are the format-fallback class: the output boundary rejected the completion in the baseline, so the case fell back to a canned refusal and now publishes a cited answer. Twenty-one of those twenty-seven carried a flagged judge check in the baseline as well (b2-de-bm18-04, b2-de-bm21-03, b2-de-hm-04, b2-de-j14-02, b2-de-q7-01, b2-de-q7-02, b2-de-rd-01, b2-de-sk-01, b2-de-wn-04, b2-jdih-a1-ret-0048, b2-jdih-mh-0005, b2-seed-007, b2-seed-025, b2-seed-037, b2-seed-040, b2-seed-041, b2-seed-044, b2-seed-045, b2-seed-047, b2-seed-085, b2-seed-086); the other six did not (b2-de-lr-01, b2-de-ls-01, b2-de-rl-01, b2-de-sb-01, b2-de-sd-02, b2-seed-018). The remaining five gains are two published answers whose sole flagged check has cleared (b2-seed-004, b2-sheet-C-1921-sum), one over_refusal that now answers (b2-bps-002), one unresolved_citation fallback that now publishes a cited answer (b2-de-wn-03) and one coverage_incomplete refusal that now answers (b2-seed-011).
Pass → fail against the baseline (8): b2-de-j25-01, b2-de-q7-03 and b2-new-nonmuslim are the three coverage_incomplete losses (they answered in the baseline); b2-de-r-04 and b2-seed-052, both expected refusals, now fail on a refusal ruling below the confidence threshold where the baseline rejected their completions outright; b2-seed-015 is the route failure; b2-de-pg-01 now carries a figures flag beside a support flag, and the rule only clears a lone figures flag; and b2-seed-030 is a completion the output boundary still rejects, now with figures and coverage flags. b2-de-r-05, b2-de-rd-02, b2-seed-043 and b2-seed-048 also fail here but were already failing in the baseline, so they are not flips against it.
The two flip sets share one shape: the answer layer moved in both directions around the corpus's coverage_incomplete refusals and the judge's flagged checks, with no single dominant mechanism in this build range. The one movement traced to a named change is b2-seed-015: #952 rewrote the search_documents description to ask for short, focused queries instead of one long conjunction, and on this question the agent now answers in one call without searching at all.
What limits comparability
- Judge noise floor. Re-judging one run's identical stored answers twice with the same judge at temperature 0 flipped 6 of 94 cases (measured 2026-08-13). A single run cannot resolve a difference of a few cases, so the small flip sets above are directional, not exact.
- The two records' frozen judge configurations differ in one field, by design. The judge model (
jev-1.13.0), its questions, equivalences, rubric,mapping_version, confidence threshold,noul_yes_thresholdand context policy are identical between the 17 and 18 October runs.adapter_sha256differs: that hash covers the adapter source, and this build's adapter source carriesjudge_flags()— the rule change itself.scripts/compare_golden_seed.pyrequires full record equality and therefore refuses the pair; the flip lists above are read from the two records' per-case verdicts instead. Every other metric in the table is a direct two-record comparison. - The scoring rule changed on 17 October 2026, and it moves verdicts without moving the system. Current-rule scores use
figures_only_true; the earlier 0.80-bar figures are kept beside them in the change table. Comparing a number from before that date with one from after it compares two scorers, not two builds. The re-scored runs' records were not re-judged, so the judge's own rulings — and therefore the noise floor above — are identical on both sides. - #952 changed the search tool's contract, and the agent's routing follows it. The change rewrote the
search_documentsdescription to ask for short, focused queries.b2-seed-015shows the effect: a correctly answered, cited question with no recordedsearch_documentscall, which fails its declared route contract. One case, but the route metric is 3 cases wide, so it moves the published route number from 100% to 66.67%. - The route check reads a provenance record, not the answer.
b2-seed-015's answer is correct and cited; the check fails because the response'stool_call_audit(search calls only) is empty and the plan named no tool. A reader should treat route pass as a trace-completeness number on this run, not as an answer-quality one. - The rule is unproven on a confident wrong number. Its evidence is a sample whose wrong answers are 57 wrong refusals and 3 assertive wrong statements, so "zero wrong answers passed" says nothing about a fluent wrong figure that the figures check endorses. See Scoring rule.
- The 16 October run is not like-for-like. It ran in a private lab (arm
fixed100, API on127.0.0.1:18510) on the PR #934 branch head, not on the box eval API and not on a merged commit. Its refusal-pass and route-pass numbers are lab behaviour; the merged code paths differ. - The three search-only runs changed nothing in retrieval and share the judged runs' retrieval denominator, so they are evidence of stability, not of quality.
- Live rows drift. Three BPS rows and one OpenData row read live portals; the eval environment gained the BPS MCP token on 16 October, which the 15 October baseline did not have. The BPS rows are therefore the least comparable in the set.
- The runner's context belief is inert but recorded. Every run here, including this one, ran with the harness default 4,096 tokens while the API served 32,768; the API's value is what scored the answers.
Context that is not comparable
| Evidence | Date and size | Reported result | Can it be compared directly? |
|---|---|---|---|
| Full label bank | 632 rows; no 632-case judged run | The bank is a label inventory, not a result. A prior provisional evaluation on 11 Oct covered 121 cases: rubric pass 45.45%, citation presence 80%, recall@5 70.8%, refusal pass 83.3%; 55 pass, 62 unsure, 4 real failures. | No. Different subset, date, run configuration, and judge state; not a full-bank score. |
| Client Studio sheet | 14 Oct 2026; 20 questions with manual mixed labels | 10 “Right/RIGHT”, 3 “partly Right/Partly Right”, 1 “Wrong”, and 6 “source right, response failed” or timeouts. Durations were recorded for only three rows (55/90/90 s). | No. Different questions, scope, and manual labels; it is not a judged run of the 100-question evaluation set. |
The comparison figures come from the 11 Oct evaluation baseline and the 14 Oct client sheet. They provide context, not a trend line.
Evaluation context and cost
The evaluation used public seed data; no production answers were included. The runner's environment named qwen3.8-27b with a 4,096-token context — its own default, inert over HTTP — while the API reported qwen3.8-27b serving 32,768 tokens. The answering model is the API's self-report, so the scores remain provisional.
Exact billed spend is unavailable because input-token usage was not persisted. Using the baseline's request accounting — one call per case for the answer/refusal rubric, 87 for answer-bearing recall, and one call per case plus one per answered output in each of the two faithfulness passes — the rerun issued about 545 judge requests (100 + 87 + 179 + 179). With retries disabled and a conservative ceiling of 65,536 input tokens per request, the maximum exposure at the published $0.042/M input-token price is 545 × 65,536 × $0.042 / 1,000,000 = $1.50. This is an upper bound, not an invoice; it shows the evaluation stayed below the $2 cap. Price source: TypeSafe model pricing.
Source files
- Run records:
eval/golden-seed/easy-runs/golden-seed-easy-20261017-172913-170652-easy.json, its-faithfulness.jsonand-judge-config.json; the earlier run's records sit beside them. - Per-question tables, both languages:
scripts/generate_docs_evaluation_table.py, with the re-scoring command ineval/golden-seed/easy-runs/README.md. - The scoring rule:
judge_flags()insrc/aceh_rag/eval/jev_judge.py; the offline re-score:scripts/rescore_golden_seed.py --from-stored-checks. - Judge calibration behind the rule (200 case-runs over the two judged runs, with the hand labels and the per-rule replay): the judge-calibration report held with the 17 October task evidence,
data/aceh-judge-calibration/report.md. - Selection, row counts, and caveats:
eval/README.md#easy-baseline. - Metric definitions and null-on-unavailable rule:
eval/README.md#metrics. - Retrieval formulas:
src/aceh_rag/eval/evaluation.py:238-269. - Answer and refusal scoring:
src/aceh_rag/eval/golden_seed.py:2004-2086,6173-6205. - Expected-span matching:
src/aceh_rag/eval/golden_seed.py:1807-1815. - “Partial”, “wrong”, and “unavailable” grades:
src/aceh_rag/eval/eval_verdicts.py:8-36. - Faithfulness formulas and fail-closed handling:
src/aceh_rag/eval/faithfulness.py:101-140,209-367. - Latency percentile interpolation:
src/aceh_rag/eval/report_breakdowns.py:75-82.