Skip to main content

Periodic Policy Updates

The Government Policy Guard is not a fixed artefact. Language moves, new evasions appear, and the line a government wants drawn can shift. This page documents the periodic policy-update process — the TOR/KAK Policy Filter line item that keeps the guard current — and, just as important, the discipline that stops an improvement in one place from quietly opening a hole in another.

The whole loop is anchored on one command:

make eval-policy

which measures the guard's catch rate and false-refusal rate against the labelled adversarial corpus (eval/policy_guard_adversarial.json) via run_policy_eval (src/aceh_rag/evaluation.py).

Why a process, and not just "edit the rules"​

The guard's rules are the lexicon and construction lists themselves (see the rules-customization surface). That makes every edit powerful and blunt: widening a rule to catch one more evasion routinely refuses a class of legitimate answers, and narrowing it to stop a false refusal opens an evasion. In a province where questions about religion, ethnicity and Qanun implementation are the day job, a careless widening is not a cosmetic regression — it silently starts refusing the legitimate work.

So a rule is never shipped on judgement alone. It is shipped through the loop below, which forces the trade-off to be measured against real labelled cases before it lands.

The loop​

  1. Observe. A new evasion slips through, or a legitimate answer is wrongly refused — from a review, a client report, or a red-team pass.
  2. Label it. Add the exact string to eval/policy_guard_adversarial.json with its correct label (inflammatory-and-should-block, or legitimate-and-should-pass) and its language (id / en / ace / mixed) and surface (/query or /generate). The corpus is the specification of what the guard must do.
  3. Let the gate fail. make eval-policy now reports a disagreement between the label and the guard's behaviour. This is the point: the run exits non-zero on any undocumented disagreement, so a new evasion or a new false refusal is a build failure, not a percentage that quietly moved. The gap is now visible and cannot be ignored.
  4. Resolve — one of two ways, both explicit:
    • Fix the rule (src/aceh_rag/policy.py) so the case is handled, then re-run make eval-policy. It passes only if the fix did not break any other labelled case — the false-refusal suite is weighted equally with the evasion suite, so a fix that catches the new evasion by refusing legitimate answers will fail the gate. This is the regression guard.
    • Accept the gap where catching one more evasion would cost a legitimate refusal. Record the case as a documented accepted limitation (with its case id) so the gate passes and the limitation is written down — never a silent miss. These accepted gaps are exactly the ones named in what the guard deliberately does not catch and in ADR-0008's limitations section.
  5. Version + ship. The classifier reports a versioned name (deterministic-rules-v2, src/aceh_rag/policy.py:69-73), so audit rows written before and after a rules change are distinguishable. Bump it when the rule set changes materially, so a reviewer can tell which guard version produced any past decision.

The result is a guard that can be improved forever without regressing: every edit is a measured trade-off against a growing set of real cases, and every accepted weakness is on the record with a case id rather than hidden in a moved number.

What still needs a human​

Two parts of "keeping the policy current" are deliberately not an engineering loop:

  • Where the line is drawn is a client decision. Deciding that Pemprov Aceh wants a particular euphemism blocked, or a particular topic left open, is the "rules customization with stakeholders" working session named in the TOR — a client-delivery activity, not something the engineering team settles unilaterally. This process gives that session a concrete, testable artefact to steer: add the agreed cases to the corpus, and the gate enforces them thereafter.
  • The corpus is ground truth only once it is validated. Every label and every line of Bahasa Indonesia and Acehnese in eval/policy_guard_adversarial.json was authored by an AI agent and prints that warning next to every rate it reports. Sign-off from a native Bahasa Indonesia speaker, a native Acehnese speaker, and a Pemprov Aceh government-communication-policy owner is required before the measured rates are an external measurement of safety rather than a regression signal on the guard's own behaviour. Onboarding those reviewers into this loop — so their judgement becomes labelled cases — is the standing task that makes the process trustworthy.