Model Serving
All inference is self-hosted for delivered environments. The app uses configurable endpoints for LLM, embedding, reranker, and OCR so local, staging, and production differ by environment, not application code.
Model roles
| Role | Current deployment profile | Production target / purpose |
|---|---|---|
| LLM | Qwen3-14B-AWQ (qwen3-14b) on the A30 staging profile | Qwen3.8-27B on 2x NVIDIA A100 40GB, tensor-parallel (TP=2); reasoning, planning, synthesis |
| Embedding | BGE-M3, 1024 dimensions, through the configured embedding endpoint | Production model remains deployment-configurable; BGE-M3 is the current contract |
| Reranker | bge-reranker-v2-m3 through the configured reranker endpoint | Cross-encoder candidate reranking |
| OCR | PaddleOCR-VL-0.9B through the GPU OCR wrapper | Text extraction from scanned/image documents |
The model id is configuration, not a hard-coded API contract. GET /models
reports the ids accepted by POST /generate; the API's /healthz endpoint
reports process liveness and active backend settings without probing model or
storage dependencies. The A30 staging deployment
template pairs a 16384-token vLLM serving window with the app's
ACEH_RAG_SYNTHESIS__SERVED_CONTEXT_WINDOW_TOKENS=16384; the Python default is
lower until an environment overrides it.
How the models are orchestrated per request
Different models run at different stages, and some run only when they are needed. The most common question — "when a PDF is uploaded, how does the system decide whether OCR is needed?" — is answered by a born-digital-first rule: the pipeline always attempts direct text extraction first and only calls the OCR model when that text is missing or unusable, which is the tell-tale of a scanned document. Image files always go straight to OCR.
Ingestion: deciding whether OCR runs
- Born-digital formats (PDF, DOCX, XLSX, PPTX, CSV, HTML, MD) are parsed for text with no model call.
- A PDF whose extraction yields no text, too few characters, or low-signal gibberish is treated as scanned and falls back to the OCR model (PaddleOCR-VL), rendering each page to an image first.
- Image files (PNG, JPG, TIFF, …) always go to the OCR model.
- Text is then chunked, and each chunk passes a gate: a model-version-comparable OCR confidence at or above the threshold (default 0.85). When the backend's own score is not production-comparable, that gate confidence is derived from the OCR output text instead. Only approved chunks are sent to the embedding model and indexed; anything below threshold is marked
needs_ocr_review, routed to a human, and excluded from retrieval (ADR-0005).
Query: retrieval, rerank, and synthesis
A query touches three configured model seams in sequence — embedding, reranker, then the LLM:
- The embedding model turns the query into a vector for semantic search; lexical search runs in parallel on the raw query text.
- Semantic (pgvector) and lexical (Postgres FTS) candidates are fused with RRF, then the reranker model orders them. If the reranker is unavailable, the fused order is used as a graceful fallback.
- The top chunks — if they fit the context budget — go to the configured LLM for synthesis. Open Data and BPS table groundings are trimmed up front to the tighter of the configured character cap and the remaining room in the served token window after the actual prompt framing, query, recalled memory, and tool directive are counted. If even the first BPS row cannot fit, the BPS read degrades to a status grounding and the Open Data fallback runs instead. A faithfulness check and the Government Policy Guard run before the cited answer is returned.
Each model above is reached through its own configured endpoint, so the per-use-case model routing described in the roadmap note below would select models at these same seams without changing the flow itself.
Runtime layout
Production sizing separates the LLM onto a tensor-parallel pair of 2x NVIDIA A100 40GB (TP=2) cards and co-locates embedding, reranking, and OCR on an A30 24 GB node. Qwen3.8-27B uses ~54 GB of FP16 weights split at ~27 GB per card, leaving roughly ~22 GB aggregate for KV cache within 80 GB aggregate VRAM. Huawei's published P3/P3snl A100-40GB line removes the former 80 GB-card availability risk; residual risk is quota and lead time for 2x A100 40GB in ap-southeast-4 (Jakarta) plus confirmation of an NVLink-connected pair. PCIe-only pairing adds token-latency overhead.
Endpoint contracts
- LLM, embedding, and reranker are called through OpenAI-compatible or app-owned HTTP contracts.
- LiteLLM is the app-facing client for model calls.
- OCR is reached through a configurable OCR service endpoint.
- Prompt templates are versioned through Langfuse and cached by the application.
Per-use-case model routing boundary
The service selects one configured model family per role for an environment.
Changing the LLM, embedding, reranker, or OCR model is a deployment-level
configuration change, not a per-request entitlement. The current staging profile
uses Qwen3-14B-AWQ, BGE-M3, bge-reranker-v2-m3, and PaddleOCR-VL-0.9B; the
production target uses the model allocation described above.
A future model-routing layer could resolve an application or use case to a dedicated endpoint and serving pool. Each model family would need its own Golden dataset evaluation, Bahasa Indonesia validation, and Government Policy Guard checks before promotion.