Skip to main content

Filter Policy Documentation

This is the client-facing documentation for the Government Policy Guard — the Policy Filter named in the TOR/KAK Policy Filter deliverable. It states honestly what the guard catches, what it deliberately does not, how the rules are shaped, and which parts an operator can change. The authoritative rationale is ADR-0008; this page is the operational read of it. Every claim here is grounded in the merged guard code (src/aceh_rag/policy_guards/policy.py) and the bank's policy slice (eval/golden-seed/labels.jsonl).

What the Policy Guard is​

The Government Policy Guard is a policy-enforcement layer that inspects every query answer before it reaches a user, and cannot be bypassed per-query (src/aceh_rag/policy_guards/policy.py:1-6). It wraps two answer surfaces:

  • grounded chat, POST /query (src/aceh_rag/services.py:1184),
  • streamed chat, POST /query/stream.

The ungrounded POST /generate primitive is an authenticated raw model call and is explicitly outside the guard (ADR-0050).

On the answer surfaces it runs on generated output only — it does not inspect the user's question (docs/adr/0008-government-policy-guard.md:82). The line it enforces is not "this topic is sensitive" but "this output advocates harm or leaks the system's own instructions."

Detection is deterministic word rules, not a model judge (src/aceh_rag/policy_guards/policy.py:7). A machine-learning judge is a real future option and ADR-0008 deliberately defers rather than rules it out; the current guard is fast (no model call, no added latency) and fully inspectable, which is why a government operator can read and edit every rule.

The classifier reports a versioned name, deterministic-rules-v2 (src/aceh_rag/policy_guards/policy.py:69-73), so that audit rows written before and after a rules change are distinguishable — the same version-drift discipline OCR extraction uses.

The governing principle: legitimate questions are protected first​

In Aceh, questions about religion, ethnicity and Qanun implementation are the day job (src/aceh_rag/policy_guards/policy.py:16-22). A civil servant asking "Apa kata Qanun Jinayat tentang non-Muslim?", an analyst pulling population statistics by religion, and a drafter working on house-of-worship permits must all get answers.

So the guard is built on one rule that governs every design choice: where catching one more evasion would cost a legitimate refusal, the evasion is accepted (src/aceh_rag/policy_guards/policy.py:20-21). Over-blocking does not make the platform safer — it makes it useless to exactly the users it exists for, and a refused legitimate question is a visible, repeated failure where a missed evasion is a rare one.

The anti-SARA rule​

SARA — Suku, Agama, Ras, Antar-golongan (ethnicity, religion, race, inter-group) — is an Indonesia-specific requirement. The guard flags SARA content only when two signals co-occur inside a bounded proximity window (src/aceh_rag/policy_guards/policy.py:227-247). Bare co-occurrence of a group name and a hostile word is not enough; an incitement construction is required.

Signal 1 — a protected-group reference​

Either a named group, or a generic group noun carrying a qualifier (src/aceh_rag/policy_guards/policy.py:132-163):

  • Named religions, denominations, adherents, ethnic and racial references — and the slurs standing in for them — in Bahasa Indonesia and English (src/aceh_rag/policy_guards/policy.py:133-146). Bare province/geography words (aceh, Jawa as in Jawa Barat) are deliberately excluded — they carry no group meaning on their own and would fire on ordinary administrative text.
  • Qualified generic nouns — kelompok minoritas, golongan tertentu, suku Jawa, religious minority (src/aceh_rag/policy_guards/policy.py:149-164). Bare kelompok/golongan are administrative words in this corpus (kelompok kerja, golongan III) and are not a signal. The capitalised-ethnonym rule (suku
    • a proper noun) is what separates suku Jawa from suku cadang (spare parts) and suku bunga (interest rate).

Signal 2 — an incitement construction​

A phrase that advocates something against people, rather than a hostile word on its own, in five families (src/aceh_rag/policy_guards/policy.py:165-218):

  1. Bahasa Indonesia advocacy — a deontic marker (harus, wajib, sebaiknya) within two tokens of a hostile action or a removal phrase (src/aceh_rag/policy_guards/policy.py:166-190).
  2. Bahasa Indonesia imperative aimed at people (usir mereka) (src/aceh_rag/policy_guards/policy.py:192-193).
  3. Denial-of-belonging euphemism in both languages (tidak layak tinggal, have no place) (src/aceh_rag/policy_guards/policy.py:195-210).
  4. English advocacy (must be expelled, should leave) (src/aceh_rag/policy_guards/policy.py:202-206).
  5. A small, best-effort Acehnese set (bèk tinggai, peuteubiet, hana layak) — see the honesty note below (src/aceh_rag/policy_guards/policy.py:211-215).

This is the difference between describing a group and advocating against one. It is what lets Pasal 28E ("warga tidak boleh diusir … karena keyakinan yang dianutnya") and a history answer about displacement during the conflict both pass while carrying the same verb diusir.

One further precision earns its keep: for removal phrased as movement, the destination decides (src/aceh_rag/policy_guards/policy.py:174-186). Removal from a place people belong to (dipindahkan keluar wilayah ini) is expulsion and is flagged; removal from a hazard (harus dipindahkan keluar zona merah, a real BPBD evacuation instruction) is not.

The bounded proximity window​

The two signals must co-occur within the incitement's own sentence plus the immediately adjacent sentences, and within 300 characters (src/aceh_rag/policy_guards/policy.py:220-224; _MAX_SENTENCE_DISTANCE = 1, _MAX_CHARACTER_GAP = 300). Both bounds defeat a specific failure:

  • A same-sentence rule (the old M0 behaviour) is beaten by a full stop — it missed "Orang Kristen tidak layak tinggal di sini. Mereka harus segera diusir."
  • A whole-document rule would refuse a long, legitimate answer that names a religion in its first paragraph and quotes a statute containing diusir nine paragraphs later.

The anti-jailbreak rule​

The anti-jailbreak rule matches constructions, not topics (src/aceh_rag/policy_guards/policy.py:279-321). Naming a subject — "what is a system prompt", "how does a jailbreak attack work" — is allowed; an answer that explains prompt security passes. What is blocked:

  • Instruction override — ignore all previous instructions, and the Bahasa imperative abaikan … instruksi (src/aceh_rag/policy_guards/policy.py:282-291). The Bahasa rule is imperative-only on purpose: matching prefixed forms such as mengabaikan would refuse every answer that describes an attack.
  • Disclosure requests — reveal the system prompt, bocorkan prompt sistem (src/aceh_rag/policy_guards/policy.py:293-300).
  • Actual disclosures in the output — here is my system prompt, berikut adalah instruksi sistem (src/aceh_rag/policy_guards/policy.py:303-309).
  • Guard bypass — bypass the policy guard, matikan filter (src/aceh_rag/policy_guards/policy.py:311-314).
  • Jailbreak activation — you are now jailbroken, mode jailbreak aktif (src/aceh_rag/policy_guards/policy.py:316-318).

Measured rates​

The historical rates below compare the hardened classifier with the old M0 classifier over the same 66 cases (36 expected block, 30 expected allow), now harvested into the bank's policy slice. Run make eval-slice SLICE=policy to check the current guard's catch rate and false refusals (ADR-0008 → Measured rates).

MetricM0 (deterministic-m0)Hardened (deterministic-rules-v2)
Catch rate (inflammatory blocked)0.222 (8/36)0.778 (28/36)
False-refusal rate (legitimate blocked)0.167 (5/30)0.000 (0/30)
Catch rate by languageid 0.200 · en 0.222 · ace 0.000 · mixed 0.667id 0.750 · en 0.889 · ace 0.750 · mixed 0.667
The policy slice is AI-authored and not yet human-reviewed

Every label, and every Bahasa Indonesia and Acehnese line in the policy slice, was authored by an AI agent (each row's label_status in eval/golden-seed/labels.jsonl says so). Until it is signed off by (a) a native Bahasa Indonesia speaker, (b) a native Acehnese speaker, and (c) an owner of Pemprov Aceh government communication policy, these rates are a regression signal on the guard's own behaviour, not an external measurement of how safe the guard is (ADR-0008 → Decision 3). The eval report prints this warning next to every number it reports (src/aceh_rag/eval/eval_tier.py:637-658). This human review is the corpus sign-off described in What still needs a human, which is a client-delivery activity, not something the engineering team decides.

What the guard deliberately does NOT catch​

ADR-0008 → What this guard still does NOT catch owns the accepted limitations, their bank case IDs and the reasons for accepting them. The policy slice reports disagreements on known_limitation rows without failing the gate; every undocumented disagreement fails it.

The rules-customization surface​

The guard's rules are the lexicon and construction lists themselves, and they are the operator-editable seam:

SurfaceLocationWhat it holds
Named protected groupssrc/aceh_rag/policy_guards/policy.py:133-146 (_GROUP_NAMED_RE)Religions, ethnicities, races, adherents, slurs
Qualified group nounssrc/aceh_rag/policy_guards/policy.py:149-164 (_GROUP_QUALIFIED_RE)kelompok minoritas, suku + ethnonym, religious minority
Incitement constructionssrc/aceh_rag/policy_guards/policy.py:165-218 (_INCITEMENT_RE and its parts)The five advocacy/removal/denial families
Jailbreak constructionssrc/aceh_rag/policy_guards/policy.py:279-321 (_JAILBREAK_RE)Override, disclosure, bypass, activation
Proximity boundssrc/aceh_rag/policy_guards/policy.py:222-223_MAX_SENTENCE_DISTANCE, _MAX_CHARACTER_GAP

What changing them costs. These lists are the whole guard, so an edit is powerful and blunt: widening a rule to catch one more evasion routinely refuses a class of legitimate answers, and narrowing it to stop a false refusal opens an evasion. This is why a rule change is never shipped on judgement alone — it is shipped through the periodic-update process, which gates every edit against the bank's policy slice with make eval-slice SLICE=policy. The PolicyClassifier seam (src/aceh_rag/policy_guards/policy.py:61-66) also lets a future model judge slot in beside the rules without a redesign, should the stakeholder session decide the accepted gaps above are worth closing.

Where the boundary is set is a client decision

Deciding where Pemprov Aceh wants the line drawn — the "rules customization with stakeholders" working session named in the TOR — is a client-delivery activity owned by humans, not something this documentation or the engineering team settles unilaterally. This page describes the surface and the cost; the client sets the value.