Deployment and Operations
The target deployment runs on Huawei Cloud AP-Southeast region with FastAPI application nodes, GPU inference nodes, PostgreSQL/pgvector, OBS, Redis, Kafka, Langfuse, APIG, and network isolation.
Cloud platform: Huawei Cloud, AP-Southeast region
Core stack: FastAPI backend, self-hosted vLLM model serving, PostgreSQL + pgvector, OBS, Redis, Kafka, and Huawei networking services
Production footprint
Production is sized for 12–18 concurrent users, with 20–25 max sustainable users and 30–40 burst users.
| # | Component | Specification | Quantity |
|---|---|---|---|
| 1 | ECS GPU — LLM inference | 16 vCPU / 64 GB RAM / 2x NVIDIA A100 40 GB (TP=2) for Qwen3.8-27B FP16, tensor-parallel across the pair | 1 |
| 2 | ECS GPU — embedding, reranker, OCR | 8 vCPU / 32 GB RAM / 1x A30 24 GB | 1 |
| 3 | ECS app servers | 4 vCPU / 16 GB RAM | 2 |
| 4 | RDS PostgreSQL + pgvector | 8 vCPU / 32 GB RAM / 1 TB SSD + backup | 1 |
| 5 | OBS object storage | ~1.2 TB | — |
| 6 | DCS Redis | 4 GB HA pair | 1 |
| 7 | DMS Kafka | Basic tier, 3 brokers | 1 |
| 8 | VPC and security groups | 1 VPC, 3 subnets | 1 |
| 9 | ELB load balancer | Shared | 1 |
| 10 | APIG API Gateway | Pay-per-use | 1 |
| 11 | NAT Gateway | Small | 1 |
| 12 | SWR container registry | Basic | 1 |
Production GPU allocation
| GPU | Hosted models | Estimated VRAM use |
|---|---|---|
| 2x NVIDIA A100 40 GB (TP=2) | Qwen3.8-27B LLM, tensor-parallel across the pair | ~54 GB total / 80 GB aggregate (~27 GB/card weights; ~22 GB aggregate KV cache) |
| A30 24 GB | Qwen3-Embedding-8B, bge-reranker-v2-m3, PaddleOCR-VL-1.5 | ~9 / 24 GB |
The production LLM uses Qwen3.8-27B with ~54 GB of FP16 weights split at ~27 GB per card across 80 GB aggregate VRAM, leaving roughly ~22 GB aggregate for KV cache. Huawei's published P3/P3snl A100-40GB line removes the former 80 GB-card availability risk. Residual risk is quota and lead time for 2x A100 40GB in ap-southeast-4 (Jakarta) and confirmation that the pair is NVLink-connected for tensor parallelism; PCIe-only pairing adds token-latency overhead.
Staging footprint
The current review staging environment is an as-built single-host stack, not the production managed-service topology. One A30 GPU host runs the API, workers, model services, PostgreSQL/pgvector, MinIO, Redis, Kafka, and Langfuse in containers.
It validates the complete AI pipeline:
LLM → embedding → vector search → rerank → OCR
It is intended for sample datasets and functional validation, not production concurrency or availability.
| # | Component | Current staging deployment | Difference from production |
|---|---|---|---|
| 1 | ECS GPU host | A30 24 GB; all staging containers run on this host | Production separates app and GPU tiers |
| 2 | API and workers | FastAPI and ingestion worker; API bound to loopback :8000 | Production uses two app nodes behind ELB/APIG |
| 3 | AI services | Qwen3-14B, BGE-M3, bge-reranker-v2-m3, and PaddleOCR-VL | Production uses the approved production model allocation |
| 4 | PostgreSQL + pgvector | On-host container | Production uses managed RDS |
| 5 | Object storage | On-host MinIO | Production uses OBS |
| 6 | Redis | On-host container | Production uses DCS Redis HA |
| 7 | Kafka | On-host container | Production uses DMS Kafka |
| 8 | Observability | On-host Langfuse stack | Production isolates and sizes observability services |
| 9 | HTTPS ingress | Caddy on public :443, forwarding to loopback :8000 | Production uses APIG and ELB |
Staging HTTPS ingress
The review API is reachable at:
https://rag-aceh-ai-eng.reviewstagepro.com
The ingress terminates TLS and forwards requests to the loopback-only FastAPI service. Port 80, the API's internal :8000, model ports :8001–:8004, and all data-service ports remain closed externally.
Every data route still requires X-API-Key. /healthz, /openapi.json, and the non-production /docs UI remain unauthenticated for discovery. Browser applications must use a BFF or gateway and must not receive the shared key.
Staging does not use an ELB or APIG. This is acceptable for review traffic, but it creates a single-host failure boundary and does not yet provide gateway rate limiting. The production topology above remains the target.
Staging GPU allocation
| Component | Serving profile | Process budget |
|---|---|---|
| Qwen3-14B-AWQ LLM | INT4, vLLM, 4k context | ~12 GB |
| BGE-M3 embedding | FP16, TEI | ~2.8 GB |
| bge-reranker-v2-m3 | FP16, Infinity | ~1.5 GB |
| PaddleOCR-VL | BF16/FP16, vLLM backend | ~3 GB |
| CUDA contexts | Four GPU processes | ~1.5 GB |
| Total in use | Single-stream staging profile | ~18.8 / 24 GB |
The GPU is the most critical staging dependency. Without it, the team cannot validate LLM inference, embeddings, reranking, or OCR under the deployed architecture.
Operational gotchas
- Bad OCR text must be gated before embedding to avoid retrieval poisoning.
- OCR model-version drift makes confidence scores incomparable unless versioned and recalibrated.
- Concurrent vector writes need serialization or transactional safety.
- Context-window overflow must fail or trim explicitly, never silently truncate.
- Documents are untrusted input, so prompt hygiene and Policy Guard are mandatory.