Skip to main content

Periodic Policy Updates

The Government Policy Guard is not a fixed artefact. Language moves, new evasions appear, and the line a government wants drawn can shift. This page documents the periodic policy-update process — the TOR/KAK Policy Filter line item that keeps the guard current — and, just as important, the discipline that stops an improvement in one place from quietly opening a hole in another.

The whole loop is anchored on one command:

make eval-slice SLICE=policy

which measures the guard's catch rate and false-refusal rate against the bank's policy slice (eval/golden-seed/labels.jsonl) through the deterministic guard (src/aceh_rag/eval/eval_tier.py).

Why a process, and not just "edit the rules"​

The guard's rules are the lexicon and construction lists themselves (see the rules-customization surface). That makes every edit powerful and blunt: widening a rule to catch one more evasion routinely refuses a class of legitimate answers, and narrowing it to stop a false refusal opens an evasion. In a province where questions about religion, ethnicity and Qanun implementation are the day job, a careless widening is not a cosmetic regression — it silently starts refusing the legitimate work.

So a rule is never shipped on judgement alone. It is shipped through the loop below, which forces the trade-off to be measured against real labelled cases before it lands.

The loop​

  1. Observe. A new evasion slips through, or a legitimate answer is wrongly refused — from a review, a client report, or a red-team pass.
  2. Label it. Add the exact string to the bank's policy slice (eval/golden-seed/labels.jsonl; selection is defined in Evaluation → Slices) with its correct label (policy_expected: block or allow) and its language (id / en / ace / mixed). Historical direct-generation- shaped cases remain classifier regressions, but runtime enforcement is on the query answer surfaces only (ADR-0050). The slice is the specification of what the guard must do where it is applied.
  3. Let the gate fail. make eval-slice SLICE=policy now reports a disagreement between the label and the guard's behaviour. This is the point: the run exits non-zero on any undocumented disagreement, so a new evasion or a new false refusal is a build failure, not a percentage that quietly moved. The gap is now visible and cannot be ignored.
  4. Resolve — one of two ways, both explicit:
    • Fix the rule (src/aceh_rag/policy_guards/policy.py) so the case is handled, then re-run make eval-slice SLICE=policy. It passes only if the fix did not break any other labelled case — the false-refusal suite is weighted equally with the evasion suite, so a fix that catches the new evasion by refusing legitimate answers will fail the gate. This is the regression guard.
    • Accept the gap where catching one more evasion would cost a legitimate refusal. Record the case as a documented accepted limitation (with its case id) so the gate passes and the limitation is written down — never a silent miss. These accepted gaps are exactly the ones named in what the guard deliberately does not catch and in ADR-0008's limitations section.
  5. Version + ship. The classifier reports a versioned name (deterministic-rules-v2, src/aceh_rag/policy_guards/policy.py:69-73), so audit rows written before and after a rules change are distinguishable. Bump it when the rule set changes materially, so a reviewer can tell which guard version produced any past decision.

The result is a guard that can be improved forever without regressing: every edit is a measured trade-off against a growing set of real cases, and every accepted weakness is on the record with a case id rather than hidden in a moved number.

What still needs a human​

Two parts of "keeping the policy current" are deliberately not an engineering loop:

  • Where the line is drawn is a client decision. Deciding that Pemprov Aceh wants a particular euphemism blocked, or a particular topic left open, is the "rules customization with stakeholders" working session named in the TOR — a client-delivery activity, not something the engineering team settles unilaterally. This process gives that session a concrete, testable artefact to steer: add the agreed cases to the policy slice, and the gate enforces them thereafter.
  • The policy slice is ground truth only once it is validated. Every label and every line of Bahasa Indonesia and Acehnese in the bank’s policy slice was authored by an AI agent and the report prints that warning next to every rate it reports. Sign-off from a native Bahasa Indonesia speaker, a native Acehnese speaker, and a Pemprov Aceh government-communication-policy owner is required before the measured rates are an external measurement of safety rather than a regression signal on the guard's own behaviour. Onboarding those reviewers into this loop — so their judgement becomes labelled cases — is the standing task that makes the process trustworthy.