Model Serving
All inference is self-hosted for delivered environments. The app uses configurable endpoints for LLM, embedding, reranker, and OCR so local, staging, and production differ by environment, not application code.
Model roles
| Role | Model / runtime | Purpose |
|---|---|---|
| LLM | Qwen3.5-27B target, with smaller GPU profiles available for constrained environments | Reasoning, planning, synthesis |
| Embedding | BGE-M3 or Qwen3-Embedding-8B deployment profile | Query and chunk embeddings |
| Reranker | bge-reranker-v2-m3 | Cross-encoder candidate reranking |
| OCR | PaddleOCR-VL GPU profile | Text extraction from scanned/image documents |
Runtime layout
Production sizing separates the LLM onto an A100 80 GB node and co-locates embedding, reranking, and OCR on a T4 16 GB node. Staging can collapse models onto a smaller GPU footprint for sample data, but still needs a real GPU to validate the full pipeline.
Endpoint contracts
- LLM, embedding, and reranker are called through OpenAI-compatible or app-owned HTTP contracts.
- LiteLLM is the app-facing client for model calls.
- OCR is reached through a configurable OCR service endpoint.
- Prompt templates are versioned through Langfuse and cached by the application.