Deployment and Operations

The target deployment runs on Huawei Cloud in Jakarta with FastAPI application nodes, GPU inference nodes, PostgreSQL vector search, OBS, Redis, Kafka, Langfuse, an API gateway, and network isolation.

Cloud platform: Huawei Cloud
Region / availability zone: AP-Jakarta / General AZ
Core stack: FastAPI backend, self-hosted vLLM model serving, PostgreSQL vector search, OBS, Redis, Kafka, and Huawei networking services

Production procurement baseline

The table below transcribes the 14 line items in the supplied Kebutuhan Spesifikasi Infrastruktur Platform AI procurement sheet. Purchase is the billed duration or usage allowance; Qty is the resource quantity on that line item. All rows target AP-Jakarta / General AZ.

#	Service / purpose	Billing / purchase	Qty	Required specification	Use
1	GPU Nvidia A100 ECS GPU — LLM inference	Yearly / 1 year	1	x86; 1x NVIDIA A100 (2×40 GB VRAM, 80 GB total); 16 vCPUs; 128 GiB	Procurement capacity for LLM reasoning, planning, and answer synthesis via vLLM. The deployed profile uses Qwen's official FP8 weights (ADR-0049) and tensor parallelism 2 on GPU device IDs `0` and `1`; the required specification is retained as the procurement baseline.
2	GPU Nvidia A30 ECS GPU — embedding, reranker, OCR	Yearly / 1 year	1	x86; 1x NVIDIA A30 (1×24 GB VRAM); 24 vCPUs; 96 GiB	Co-locates three models: Qwen3-Embedding-8B (vector generation), bge-reranker-v2-m3 (result re-scoring), and PaddleOCR-VL (scanned document extraction).
3	Elastic Volume Service 1 GPU block storage	Yearly / 1 year	4	Ultra-high I/O; 1,000 GiB for GPU per volume	Attachable SSD storage for GPU hosts — model weight downloads, KV cache swap, and OCR scratch space.
4	Elastic Cloud Server ECS application servers	Yearly / 1 year	2	x86; General computing-plus c9.xlarge.4; 4 vCPUs; 16 GiB; AlmaLinux 9.4 64-bit; 40 GB General Purpose SSD	Run the FastAPI backend, orchestrator, and ingestion workers behind the load balancer. Paired for HA.
5	CCE+ECS+EVS+OBS Self-managed PostgreSQL + vectorchord on Percona	Yearly / 1 year	1	PostgreSQL 17; Single; Dedicated; 8 vCPUs; 32 GB; 1,000 GB Extreme SSD; vectorchord	Primary data store: document/chunk vectors, full-text lexical search, conversation history, and audit log.
6	Object Storage Service OBS object storage	Pay-per-use / 1 year	1	Standard; 20 TB; Single-AZ object storage; 12-month storage duration	Stores raw source documents (PDFs, DOCX, images), OCR extract outputs, and ingestion artefacts. Serves as the durable document corpus backing the index.
7	DCS (for Redis) Managed Redis	Yearly / 1 year	1	Basic; Redis 7.0; Master/Standby; x86; DRAM; 4 GB	Session cache, API rate limiting, distributed ingestion locks, and Kafka consumer-lag tracking. Master/Standby for failover.
8	DMS (for Kafka) Managed Kafka	Yearly / 1 year	1	kafka.2u4g.cluster.small; 3 brokers; Ultra-high I/O; 600 GB total, 200 GB per broker	Async ingestion pipeline: document event dispatch, chunk-write queuing, and retry/backpressure for the ingestion workers.
9	Reverse Proxy	Yearly / 1 year	1	s7n.large.2; 2 vCPUs; 4 GiB; High I/O; 40 GB	Provides the production reverse-proxy layer in front of the application ingress.
10	NAT Gateway	Yearly / 1 year	1	Small	Outbound internet access for private-subnet services — model weight downloads, webhook delivery, and dependency updates.
11	Web Application Firewall	Pay-per-use / 8,766 hours	1	10amounts; 1PCS; 200Million times	Shields the public API endpoint from OWASP-top-10 attacks, bot traffic, and volumetric abuse.
12	Elastic IP	Pay-per-use / 8,766 hours	2	Dedicated Dynamic BGP; 15 TB traffic per IP	Static public IPs for the NAT gateway, APIG, ELB, and WAF egress endpoints.
13	Log Tank Service	Yearly / 1 year	1	5 GB/day; 30-day retention	Centralized retention for production service and infrastructure logs.
14	Cloud Backup	Yearly / 1 year	4	1,000 GB per backup resource	Backup capacity for production data and service volumes.

Architecture mapping and confirmation gates

The procurement baseline and the application architecture are related but not interchangeable. Resolve these items before production acceptance:

The procurement wording above specifies A100 capacity and is not a statement of the deployed device topology. The production serving profile runs Qwen3.8-27B from the official FP8 checkpoint (Qwen/Qwen3.8-27B-FP8, weight-only on A100; ADR-0049) with tensor parallelism 2 across GPU device IDs 0 and 1 (see docker-compose.models.prod.inference.yml and docs/deployment/model-serving-prod.md).

The production database is self-managed PostgreSQL 17 with vectorchord on Percona on the CCE+ECS+EVS+OBS compute and storage components; the procurement specification is Single, Dedicated, with 8 vCPUs, 32 GB, and 1,000 GB Extreme SSD.
The four EVS volumes provide 4,000 GiB aggregate GPU-attached capacity if each 1,000 GiB requirement is provisioned as written.

Staging footprint (historical — staging torn down)

Historical. The review-staging hosts are torn down (firstmate production audit 2026-10-07). Production serves deployed ref 2cc9a417 on apps-0002 with the client frontend at https://rag-aceh-neo.reviewstagepro.com (Studio at /studio). What follows describes the former staging box, not current state.

The current review staging environment is an as-built single-host stack, not the production managed-service topology. One A30 GPU host runs the API, workers, model services, PostgreSQL/pgvector, MinIO, Redis, Kafka, and Langfuse in containers.

It validates the complete AI pipeline:

LLM → embedding → vector search → rerank → OCR

It is intended for sample datasets and functional validation, not production concurrency or availability.

#	Component	Current staging deployment	Difference from production
1	ECS GPU host	A30 24 GB; all staging containers run on this host	Production separates app and GPU tiers
2	API and workers	FastAPI and ingestion worker; API bound to loopback `:8000`	Production uses two app nodes behind ELB/APIG
3	AI services	Qwen3-14B, BGE-M3, bge-reranker-v2-m3, and PaddleOCR-VL	Production uses the approved production model allocation
4	PostgreSQL + pgvector	On-host container	Production uses self-managed PostgreSQL 17 with vectorchord on Percona on the CCE+ECS+EVS+OBS compute and storage components
5	Object storage	On-host MinIO	Production uses OBS
6	Redis	On-host container	Production uses DCS Redis HA
7	Kafka	On-host container	Production uses DMS Kafka
8	Observability	On-host Langfuse stack	Production isolates and sizes observability services
9	HTTPS ingress	Caddy on public `:443`, forwarding to loopback `:8000`	Production uses APIG and ELB

Staging HTTPS ingress (historical — torn down)

The review API was reachable at:

https://rag-aceh-ai-eng.reviewstagepro.com

The ingress terminates TLS and forwards requests to the loopback-only FastAPI service. Port 80, the API's internal :8000, model ports :8001–:8004, and all data-service ports remain closed externally.

Every data route still requires X-API-Key. /healthz, /readyz, /openapi.json, and the non-production /docs UI remain unauthenticated for discovery and orchestration probes. Browser applications must use a BFF or gateway and must not receive the shared key.

Staging does not use an ELB or APIG. This is acceptable for review traffic, but it creates a single-host failure boundary and does not yet provide gateway rate limiting. The production topology above remains the target.

Staging GPU allocation

Component	Serving profile	Process budget
Qwen3-14B-AWQ LLM	INT4, vLLM, 4k context	~12 GB
BGE-M3 embedding	FP16, TEI	~2.8 GB
bge-reranker-v2-m3	FP16, Infinity	~1.5 GB
PaddleOCR-VL	BF16/FP16, vLLM backend	~3 GB
CUDA contexts	Four GPU processes	~1.5 GB
Total in use	Single-stream staging profile	~18.8 / 24 GB

The GPU is the most critical staging dependency. Without it, the team cannot validate LLM inference, embeddings, reranking, or OCR under the deployed architecture.

External MCP server configuration

The application has a strict, disabled-by-default registry and request-local integration seam for approved remote MCP servers. Settings loading never opens a network connection. When both gates are enabled, the request builds an HTTPS Streamable HTTP client behind an Aceh-owned wrapper; a raw MCP toolset is never attached to the agent. No client endpoint is registered or enabled yet. The staged rollout is tracked by RFC #683.

ACEH_RAG_MCP__ENABLED is the global kill switch and defaults to false. ACEH_RAG_MCP__SERVERS is a JSON object keyed by stable server ID. Every entry also has its own enabled flag, which defaults to false. A reviewed entry has this shape:

{
  "satu_data_pilot": {
    "enabled": false,
    "transport": "streamable_http",
    "url_env": "ACEH_RAG_MCP_SATU_DATA_URL",
    "auth_secret_env": "ACEH_RAG_MCP_SATU_DATA_TOKEN", // pragma: allowlist secret
    "tool_prefix": "satu_data_mcp",
    "allowed_tools": ["search_datasets", "read_dataset"],
    "expected_schema_digests": {
      "search_datasets": "sha256:<64-lowercase-hex>",
      "read_dataset": "sha256:<64-lowercase-hex>"
    },
    "access_mode": "public_unscoped",
    "require_provenance": true,
    "max_calls_per_question": 4,
    "timeout_seconds": 20,
    "max_result_bytes": 262144
  }
}

ACEH_RAG_MCP__VISIBILITY_RULES is a strict deployment-managed list of Organization assignments. Each assignment names only servers and exact tools already present in the reviewed registry; duplicate Organizations/servers and unknown server/tool references stop settings loading. This synthetic example shows the shape without inventing the client's still-pending canonical Organization codes:

[
  {
    "organization": "org_code_placeholder",
    "sources": [
      {
        "server_id": "satu_data_pilot",
        "tool_ids": ["search_datasets", "read_dataset"]
      }
    ]
  }
]

For a verified profile, Echelon 0 and 1 receive every enabled reviewed server/tool by default. Echelons 2, 3, 4, and 99 receive only their Organization assignment. Missing or unknown profiles receive no MCP tools. POST /query and /query/stream accept an optional mcp_source_ids list: omission uses the profile default or the conversation's saved choice, an empty list disables every eligible source, and a non-empty list may only reduce the eligible set. The saved choice is scoped to the owning conversation. An ineligible or unknown source is rejected before discovery, and the immutable grant is checked again before execution.

The client's current MCP endpoints expose public data and do not filter on opd_id or eselon_id; these headers are future-compatible metadata, not authorization. Therefore this application-side policy is the pilot's only functional access boundary. Wiring the policy to real verified claims and forwarding those formal headers belongs to #699; until then, an unverified profile receives no MCP tools. Collection/document source selections continue to withhold MCP tools completely.

The registry stores only the names of deployment-provided URL and credential variables, never their values. When an entry is enabled, its referenced URL must be an HTTPS Streamable HTTP endpoint and its credential must be non-empty. The resolved values stay out of settings serialization and validation errors. Stdio commands, legacy SSE transports or /sse endpoints, unknown fields, duplicate prefixes, unlisted tools, schema-digest gaps, future ABAC modes, and out-of-range limits fail configuration loading.

Schema digests use sha256: followed by exactly 64 lowercase hexadecimal characters, and their keys must exactly match allowed_tools. The digest input is canonical compact JSON with sorted keys in this exact shape:

{"input_schema": <discovered inputSchema>, "output_schema": <discovered outputSchema or null>}

The wrapper exposes a tool only when its exact name is allow-listed, this digest matches, and the discovered definition explicitly marks it read-only and non-destructive. It adds the configured stable prefix, discards server-authored descriptions and metadata, and never accepts server instructions, resources, prompts, roots, sampling, elicitation, or task execution. Unknown tools and schema drift are withheld and produce bounded operator diagnostics.

Every accepted call spends the shared request budget and its server-specific budget. The wrapper applies repeat and placeholder protection, time and result-size limits, progress events, untrusted-text neutralisation, and the existing grounding ceiling before recording the exact model-visible evidence in the request passage ledger. A public_unscoped server is not constructed when a collection or document source boundary is active.

MCP evidence contract

An MCP result can support a factual answer only when the server returns this envelope as MCP structuredContent:

{
  "evidence": [
    {
      "text": "Exact source evidence shown to the model",
      "source_title": "Dataset or record title",
      "source_name": "Publishing organisation",
      "source_id": "stable-source-id",
      "source_url": "https://data.example.id/source/stable-source-id",
      "retrieved_at": "2026-09-22T00:00:00Z",
      "period": "2025",
      "version": "dataset-version-if-available"
    }
  ]
}

text, source_title, source_name, source_id, source_url, and retrieved_at are required. period and version are optional. The retrieval time must include a timezone, the source URL must use HTTPS without embedded credentials, and one result may contain at most 20 evidence items. Unknown fields are rejected.

The whole envelope is validated before any passage is recorded. Plain text, legacy structured results, partially valid arrays, missing source identity, unsafe URLs, schema drift, and over-limit evidence lists therefore fail closed and cannot ground an answer. Each accepted item becomes a separate post-budget passage and carries the stable MCP server/tool/call plus source identity through the normal faithfulness gate, public tool citation, claim map, provenance trace, and persisted message. External text and display metadata remain untrusted and cross the same neutralisation and length boundaries as native portal evidence.

The only accepted access mode in this first slice is public_unscoped; speculative ABAC modes remain invalid configuration.

Operational gotchas

Bad OCR text must be gated before embedding to avoid retrieval poisoning.
OCR model-version drift makes confidence scores incomparable unless versioned and recalibrated.
Concurrent vector writes need serialization or transactional safety.
Context-window overflow must fail or trim explicitly, never silently truncate.
Documents are untrusted input, so prompt hygiene and Policy Guard are mandatory.

Production procurement baseline​

Architecture mapping and confirmation gates​

Staging footprint (historical — staging torn down)​

Staging HTTPS ingress (historical — torn down)​

Staging GPU allocation​

External MCP server configuration​

MCP evidence contract​

Operational gotchas​