Evaluation and Roadmap
The project validates retrieval, answer quality, citation quality, performance, OCR, and deployment behavior in stages.
Evaluation framework
| Dimension | Method |
|---|---|
| Retrieval quality | Recall@k, MRR, nDCG on the Golden dataset |
| Answer quality | LLM-as-judge and human review against expected answers |
| Faithfulness | Check every answer claim against retrieved chunks |
| Citation accuracy | Confirm answer citations point to supporting chunks |
| Performance | p50/p95/p99 latency, tokens/sec, concurrent request envelope |
| OCR quality | Confidence distribution, review rate, corrected-output acceptance |
Golden dataset
The Golden dataset is the acceptance benchmark: representative documents plus labeled query-answer pairs and source-passage mappings. If passage-level labels are not available, document-level labels are the fallback with lower precision.
Milestones
| Stage | Scope |
|---|---|
| Foundation | Single-shot retrieve to synthesize, multi-format ingestion, OCR confidence gating, hybrid retrieval, grounded answers, Policy Guard, Langfuse, and upload/status APIs |
| Agentic expansion | Router, Planner, Executor, tool registry, multi-turn memory, session resume, and query understanding |
| Governance and scale | RBAC/ABAC, full audit logging, KTP extraction, human-review workflows, source connectors, and scale hardening |
| Acceptance | Full Golden dataset, model evaluation report, dashboard data APIs, technical documentation, and training |
Validation gates
- Service-contract validation for API behavior.
- Integration validation for PostgreSQL/pgvector and object storage paths.
- End-to-end validation for ingest, OCR, retrieval, and query.
- GPU validation for LLM, embedding, OCR, and reranker.
- Production-stack evaluation against the Golden dataset before acceptance.