Skip to main content

Answer quality review

When a user dislikes an answer (PUT /messages/{message_id}/feedback with "rating": "dislike"), the answer opens for review. These three routes are the admin side of that flow. The partner admin app renders the screens; this page is the API contract.

MethodPathPurpose
GET/quality/flaggedList the answers under review, newest complaint first.
GET/quality/flagged/{message_id}One flagged answer, with its root causes and review history.
POST/quality/flagged/{message_id}/reviewSet the review status and add a note.

All three require the configured API key and a verified admin principal: a verified JWT, or the claims the trusted edge mints. operator and user get 403. With verified authentication off (require_auth), the routes return 503: a caller-asserted X-User-Role grants nothing, because these views show every user's questions and notes. This is the same gate as MCP and prompt management. Show the Quality menu to admins only. Browser clients go through a same-origin BFF and never hold the shared API key.

Each read writes an ai_engine_audit_log row (action = read, entityType = message) naming the admin, because these views show users' questions. Each review writes a review row naming the admin, together with the new status, the status it replaced, the note and the issue link. The status change and that row are saved together or not at all.

Review status​

open ──▶ reviewing ──▶ resolved
              │
              └──▶ needs_prompt_change ──▶ resolved

A dislike sets open if the answer has no status yet, and reopens it as open after an admin resolved it. It never resets a review in progress (reviewing, needs_prompt_change), and a later like never changes the status. An admin may set any of the four statuses. needs_prompt_change requires an issue_url, because prompts change only by pull request and deploy. There is no prompt editor; show the help text "Perubahan prompt melalui tim AI (issue → rilis)".

GET /quality/flagged​

Query parameterTypeMeaning
statusopen | reviewing | needs_prompt_change | resolvedReview status.
reasonwrong_answer | wrong_citation | incomplete | outdated | offensive_or_sara | otherFeedback reason.
ratinglike | dislikeLatest rating. A user may change a dislike to a like after the review opened.
fromISO 8601 date-timeFeedback at or after this time (inclusive). A time with no offset is read as UTC.
toISO 8601 date-timeFeedback before this time (exclusive).
toolstringMatches the tool that built the grounding, or any tool the agent called.
fallback_reasonstringThe recorded fallback, for example over_refusal, coverage_incomplete or no_answer.
cursorstringThe previous page's next_cursor. A cursor the API did not issue returns 422.
limit1–100, default 25Page size.

Filters combine with AND. A tool or fallback_reason containing NUL (U+0000) returns 422. Paging is by keyset on (feedback_at, message_id), so rows that arrive while a user pages are not repeated.

{
  "items": [
    {
      "message_id": "00000000-0000-0000-0000-000000000102",
      "conversation_id": "00000000-0000-0000-0000-000000000101",
      "question_preview": "Apa isi Qanun Aceh Nomor 7 Tahun 2024 tentang zakat?",
      "rating": "dislike",
      "reason": "wrong_citation",
      "review_status": "open",
      "grounding_rate": 0.78,
      "fallback_reason": null,
      "tool_executed": "vector_search",
      "feedback_at": "2026-11-03T08:00:00+00:00",
      "created_at": "2026-11-03T07:59:12+00:00"
    }
  ],
  "next_cursor": "<opaque cursor>"
}

question_preview is the first 200 characters of the user question the answer replied to. grounding_rate is the claim grounding rate (claims verified out of claims drafted, 0–1), and is null when nothing was checkable or the answer predates the rate. Show — for null, not 0%. tool_executed is null on answers stored before this field was recorded.

GET /quality/flagged/{message_id}​

{
  "message_id": "00000000-0000-0000-0000-000000000102",
  "conversation_id": "00000000-0000-0000-0000-000000000101",
  "question": "Apa isi Qanun Aceh Nomor 7 Tahun 2024 tentang zakat?",
  "answer": "Qanun tersebut mengatur ... [1]",
  "citations": [{"document_id": "…", "chunk_id": "…", "document_title": "…"}],
  "grounding_rate": 0.78,
  "claims_supported": 7,
  "claims_total": 9,
  "feedback": {
    "rating": "dislike",
    "reason": "wrong_citation",
    "note": "Pasal yang dikutip salah",
    "at": "2026-11-03T08:00:00+00:00"
  },
  "review_status": "reviewing",
  "review_history": [
    {
      "actor": "11111111-1111-4111-8111-111111111111",
      "action": "review",
      "status": "reviewing",
      "rating": null,
      "reason": null,
      "note": "Cek ulang pasal",
      "issue_url": null,
      "at": "2026-11-03T09:10:00+00:00"
    }
  ],
  "root_causes": [
    {
      "code": "mis_citation",
      "label_id": "Sitasi tidak valid",
      "label_en": "Claim cited a passage it was not shown",
      "evidence": {"fabricated_label": 1}
    }
  ],
  "trace_available": true,
  "created_at": "2026-11-03T07:59:12+00:00"
}

answer and citations are what the user was shown, after the Government Policy Guard. review_history lists the answer's feedback and review audit events, oldest first.

Root causes​

Root causes are read from what the answer recorded: its message metadata and its per-answer trace (ADR-0046). Each one is a check on a stored count, code or flag, not a model judgment, and evidence holds only those counts and codes, never answer or claim text. They point a reviewer at a stage; they do not decide whether the answer was wrong.

codeFires whenevidence
retrieval_missThe fallback was no_answer (below the no-answer floor), or the trace shows retrieval ran and returned no candidates.fallback_reason, no_answer, or retrieval_legs + candidates
over_refusalThe fallback was over_refusal or coverage_incomplete.fallback_reason
mis_citationA claim cited a passage label it was not shown (/query/v2: an invalid_citation verdict).fabricated_label or invalid_citation count
unsupported_claimsThe support check rejected a claim (/query/v2: an unsupported verdict).unsupported or v2_unsupported count
policy_guard_blockThe Policy Guard blocked the answer.action, and categories when recorded
no_issue_foundNone of the above. A human decides.{}

Several causes can fire together; they are listed in the order of the table. When the trace row is missing, or was written in a format this version cannot read, trace_available is false and root causes come from the message metadata alone.

An unknown, malformed or never-flagged message_id returns 404.

POST /quality/flagged/{message_id}/review​

{
  "status": "needs_prompt_change",
  "note": "Instruksi sitasi perlu diperjelas",
  "issue_url": "https://github.com/insourcedata/superfk-aceh/issues/1234"
}
FieldRule
statusRequired; one of the four review statuses.
noteOptional; at most 1000 characters; no NUL (U+0000).
issue_urlOptional http(s) URL with no NUL (U+0000); required when status is needs_prompt_change.

Unknown fields, an invalid status, a long note, a non-http(s) link, a NUL in the note or link, or a missing link for needs_prompt_change return 422, and nothing changes. The response is the updated detail, the same shape as GET /quality/flagged/{message_id}.

Errors​

StatusMeaning
401Missing or invalid API key, or a required verified principal is absent.
403The verified principal is not an administrator.
404The answer is unknown, malformed or not under review.
422An invalid filter, cursor or review body.
503Verified authentication is off, so the review surface is unavailable.

Not in this version​

  • A summary dashboard endpoint (counts by status, reason, root cause). The list's filters cover the review flow; it will be added if the partner asks.
  • An AI-written root-cause summary. It would need a model judge and a Bahasa spot-check (ADR-0037).