Evaluation and Roadmap
The project validates retrieval, answer quality, citation quality, performance, OCR, and deployment behavior in stages.
Evaluation framework
| Dimension | Method |
|---|---|
| Retrieval quality | Recall@k, MRR, nDCG on the Golden dataset |
| Answer quality | LLM-as-judge and human review against expected answers |
| Faithfulness | Check every answer claim against retrieved chunks |
| Citation accuracy | Confirm answer citations point to supporting chunks |
| Performance | p50/p95/p99 latency, tokens/sec, concurrent request envelope |
| OCR quality | Confidence distribution, review rate, corrected-output acceptance |
Golden dataset
The Golden dataset is the acceptance benchmark: representative documents plus labeled query-answer pairs and source-passage mappings. If passage-level labels are not available, document-level labels are the fallback with lower precision.
Initial results
These are early, indicative numbers from live golden-seed runs against a local pgvector index and staging models (a smaller staging LLM over an SSH tunnel), not the final acceptance report. They validate the retrieval and chunking direction; absolute figures are expected to improve on the production 27B model.
- Hybrid retrieval on the seed set — Recall@5 ≈ 0.83, MRR ≈ 0.80, nDCG@5 ≈ 0.81 across the mixed JDIH + PPID labels.
- Structure-aware chunking beats a flat baseline on legal text. On born-digital JDIH, a Pasal-aware chunker (article-boundary chunking) outperformed vanilla fixed-window chunking on the same corpus — retrieval and faithfulness both moved up (roughly +0.04 retrieval, +0.05 faithfulness in the A/B). PPID prose is validated on the layout-block chunker, with correct top source retrieved on ~95% of PPID labels; a PPID structure-aware chunker is a merged candidate still in A/B evaluation.
- Figure-aware faithfulness reduces false refusals. After making the support check figure-aware (exact numeric/currency/percent/ratio grounding across Indonesian and English), previously-refused terse and cross-language figure answers ground correctly without weakening the gate against unsupported claims.
The remaining answer-level misses in these runs were not retrieval or chunking failures — they traced to configured-source routing (a "how many" question sent to an unconfigured SQL source) and a small number of scanned budget tables blocked on the OCR path, both tracked separately.
Milestones
| Stage | Scope |
|---|---|
| Foundation | Single-shot retrieve to synthesize, multi-format ingestion, OCR confidence gating, hybrid retrieval, grounded answers, Policy Guard, Langfuse, and upload/status APIs |
| Agentic expansion | Router, Planner, Executor, tool registry, multi-turn memory, session resume, and query understanding |
| Governance and scale | RBAC/ABAC, full audit logging, KTP extraction, human-review workflows, source connectors, and scale hardening |
| Acceptance | Full Golden dataset, model evaluation report, dashboard data APIs, technical documentation, and training |
Validation gates
- Service-contract validation for API behavior.
- Integration validation for PostgreSQL/pgvector and object storage paths.
- End-to-end validation for ingest, OCR, retrieval, and query.
- GPU validation for LLM, embedding, OCR, and reranker.
- Production-stack evaluation against the Golden dataset before acceptance.