Model Serving

All inference is self-hosted for delivered environments. The app uses configurable endpoints for LLM, embedding, reranker, and OCR so local and production differ by environment, not application code. (Former review-staging hosts are torn down — firstmate production audit 2026-10-07.)

Model roles

Role	Local profile	Purpose
LLM	Qwen3-14B-AWQ (qwen3-14b) on the A30 profile	Reasoning, planning, synthesis
Embedding	BGE-M3, 1024 dimensions	Query and document vectors
Reranker	bge-reranker-v2-m3	Cross-encoder candidate reranking
OCR	PaddleOCR-VL-0.9B through the OCR wrapper	Scanned/image text extraction

The model id is configuration, not a hard-coded API contract. GET /models reports the ids accepted by POST /generate; /healthz reports process liveness while /readyz probes the embedding and reranker health endpoints alongside the required data and observability services. Production serves a 65536-token window (ACEH_RAG_SYNTHESIS__SERVED_CONTEXT_WINDOW_TOKENS=65536 in ansible/templates/env.prod.j2); the Python default is lower until an environment overrides it.

How the models are orchestrated per request

Different models run at different stages, and some run only when they are needed. The most common question — "when a PDF is uploaded, how does the system decide whether OCR is needed?" — is answered by a born-digital-first rule: the pipeline always attempts direct text extraction first and only calls the OCR model when that text is missing or unusable, which is the tell-tale of a scanned document. Image files always go straight to OCR.

Ingestion: deciding whether OCR runs

• Born-digital formats (PDF, DOCX, XLSX, PPTX, CSV, HTML, MD) are parsed for text with no model call.

• A PDF whose extraction yields no text, too few characters, or low-signal gibberish is treated as scanned and falls back to the OCR model (PaddleOCR-VL), rendering each page to an image first.

  A NUL in native PDF text also triggers this fallback when an OCR client is configured and earlier routing has not already returned a document. Assembly preserves that signal as ParsedDocument.text_had_nul even though the text is already cleaned; a NUL only in metadata does not trigger OCR. Ingestion and extract_ocr share this decision. Without an OCR client, the cleaned native text remains available. Regression: tests/unit/test_dirty_ocr_layer_detection.py::test_native_text_nul_preserves_ocr_fallback_after_sanitization.

  A text layer alone does not establish that a PDF is born-digital. The parser's is_stamp_only_text_layer check removes normalized lines occurring on more than half of a multi-page PDF's pages, then checks whether the remaining non-whitespace characters average less than MIN_TEXT_LAYER_CHARS_PER_PAGE (50) per page. Empty layers and scanner-app stamps therefore route the whole PDF to OCR when an OCR client is configured, before the table route. This structural check also routes sparse born-digital PDFs below the threshold to OCR; that extra OCR cost is an accepted tradeoff. With no OCR client configured, this check does not reroute the document. Regression coverage: tests/unit/test_stamp_only_text_layer.py.

• Image files (PNG, JPG, TIFF, …) always go to the OCR model.

• Text is then chunked, and each chunk passes a gate: a model-version-comparable OCR confidence at or above the threshold (default 0.85). When the backend's own score is not production-comparable, that gate confidence is derived from the OCR output text instead. Only approved chunks are sent to the embedding model and indexed; anything below threshold is marked needs_ocr_review, routed to a human, and excluded from retrieval (ADR-0005).

Query: retrieval, rerank, and synthesis

A query touches three configured model seams in sequence — embedding, reranker, then the LLM:

• The embedding model turns the query into a vector for semantic search; lexical search runs in parallel on the raw query text.
• Semantic (pgvector) and lexical (Postgres FTS) candidates are fused with RRF, then the reranker model orders them. If the reranker is unavailable, the fused order is used as a graceful fallback.
• The top chunks — if they fit the context budget — go to the configured LLM for synthesis. Open Data and BPS table groundings are trimmed up front to the tighter of the configured character cap and the remaining room in the served token window after the actual prompt framing, query, recalled memory, and tool directive are counted. If even the first BPS row cannot fit, the BPS read degrades to a status grounding and the Open Data fallback runs instead. A faithfulness check and the Government Policy Guard run before the cited answer is returned.

Each model above is reached through its own configured endpoint, so the per-use-case model routing described in the roadmap note below would select models at these same seams without changing the flow itself.

Runtime layout

See the audited production snapshot for the deployed ref, runtime settings, corpus inventory, and pending deployments.

Endpoint contracts

• LLM, embedding, and reranker are called through OpenAI-compatible or app-owned HTTP contracts.
• LiteLLM is the app-facing client for model calls.
• OCR is reached through a configurable OCR service endpoint.
• Prompt templates are file-versioned in the repo; only the agent's agent-editorial role/audience/language fragment may be fetched from Langfuse behind its code-owned contract (ADR-0006). Editing any other prompt in Langfuse has no runtime effect.

Per-use-case model routing boundary

Current boundary

The service selects one configured model family per role for an environment. Changing the LLM, embedding, reranker, or OCR model is a deployment-level configuration change, not a per-request entitlement. The local A30 profile uses the model allocation above; deployed production settings are owned by the runbook linked under Runtime layout.

A future model-routing layer could resolve an application or use case to a dedicated endpoint and serving pool. Each model family would need its own Golden dataset evaluation, Bahasa Indonesia validation, and Government Policy Guard checks before promotion.