Data Ingestion
Data ingestion prepares raw government documents for retrieval and grounded answering. It validates inputs, stores raw bytes, parses or OCRs content, extracts metadata, chunks text, embeds approved chunks, builds lexical text, persists rows, and exposes processing status.
This page explains ingestion architecture and lifecycle decisions. For request and response fields, status codes, error bodies, curl commands, and TypeScript examples, use the Ingestion and Documents API reference. Architecture summaries below are not the complete wire contract.
Supported sources
The data model recognizes five source types: JDIH, OpenData, PPID,
SatuData, and upload.
| Source | Current handling |
|---|---|
| JDIH | Legal documents follow the parse → OCR fallback → chunk → embed → index path. |
| PPID | Public-information documents follow the same ingestion path; PDF extraction follows the routing references below. |
upload | Manual or integration files use the upload and batch APIs. |
| OpenData | Contract-compatible structured events are skipped before parse/chunk/embed/index; query-time Open Data reads are the source of truth when enabled. |
| SatuData | Contract-compatible structured events are skipped before parse/chunk/embed/index; query-time Open Data reads are the source of truth when enabled. |
The structured-source skip is deliberate: embedding a portal snapshot would
create a stale competing copy. The worker records a terminal skipped outcome
for internal upload jobs and emits indexed/skipped on the wire (_TERMINAL_INTERNAL_EVENT, src/aceh_rag/kafka/kafka_ingestion.py).
The production Kafka worker was recreated on the deployed API image (2cc9a417,
firstmate production audit 2026-10-07); the worker runs the doc.uploaded
consumer contract in partner-events.
Supported formats
Docling Slim is the born-digital parser boundary for PDF, DOCX, PPTX, XLSX, CSV, MD, and HTML. TXT is handled locally. Images and scanned documents route through OCR.
For PDF scan routing, see model orchestration. For selective native-first table extraction and its OCR fallback, see ADR-0012.
Ingestion entry points
The table below names the architectural entry points and what each path is responsible for. It intentionally omits exact payloads, response bodies, status codes, and error shapes; those belong in the Ingestion and Documents API reference.
| Method | Endpoint or seam | Architecture responsibility |
|---|---|---|
| Manual multipart upload | POST /documents | Accept user or integration files, persist raw bytes, create status-tracked ingestion work, and schedule per-file processing |
| Inline text | POST /documents/txt | Index text supplied directly by a trusted caller without routing through OCR |
| Batch object-storage prefix | POST /ingestion/batch with prefix | Discover objects under a configured prefix and enqueue matched corpus objects |
| Batch manifest | POST /ingestion/batch with explicit keys | Enqueue selected object keys from the configured object store |
| API integration | Ingestion API through the FastAPI boundary | Let external integrations enter the same validation, storage, queueing, and worker path as frontend uploads |
| Scheduled ingestion | Scheduler calling API or batch connectors | Reuse the same batch/source APIs as other integrations |
| Streaming ingestion | Kafka/DMS worker processing | Support async source-specific adapters without bypassing validation and indexing rules |
End-to-end pipeline
For OCR documents, each below-threshold or lost page is excluded from indexing.
The document-level needs_review verdict uses the configured failed-page
fraction, so one weak cover page does not automatically quarantine an otherwise
usable scan; a lost page always forces review. GET /documents exposes
chunk_count, indexed_chunk_count, and parked_chunk_count so clients can
distinguish a partially retrievable review outcome from a document with no
indexed chunks.
Validation
Validation happens before storage, before parsing, and before indexing.
| Stage | Validation |
|---|---|
| Upload request | API key, supported extension/type, file size, batch size |
| Object storage | Backend errors mapped to domain errors; partial upload cleanup prevents orphans |
| Worker message | Required fields: document ID, content SHA-256, filename, source type, source version |
| Raw bytes | Archive-bomb limits run before parse/OCR |
| Parsed text | Empty documents rejected |
| Chunk text | Low-signal/gibberish content routed to review before embedding |
| OCR chunks | Threshold gate on a model-version-comparable confidence (ADR-0005) |
| Repository | Duplicate ordinals rejected; indexed chunks require embeddings; needs_ocr_review chunks are excluded from retrieval |
Metadata extraction and persistence
The ingestion model stores document-level and chunk-level metadata.
Document rows contain title, source type, format, source version, ingest timestamp, status, and metadata JSON. Chunk rows contain document ID, ordinal, text, embedding, lexical text, generated lexical_tsv, OCR confidence, chunk status, page number, section path, legal reference, character spans, and metadata JSON.
The current ingestion service records parser metadata, OCR confidence threshold, content hash, source key, OCR model, confidence comparability, and page/source span fields.
ParsedDocument assembly removes NUL characters (U+0000) from extracted text,
page/block content and parser metadata, remapping canonical and structure offsets
against their respective cleaned texts. Chunk offsets, fact anchors/unit offsets,
fact fingerprints and parsed-text snapshots therefore derive from the same NUL-free
text. Ingestion also cleans source metadata and identity keys before enrichment.
Both assembly and PostgreSQL writes use the shared helpers in
src/aceh_rag/text_sanitization.py. The repository retains a last-resort guard for
document titles, metadata, snapshots, fact updates and chunk text-bearing fields:
PostgreSQL rejects NUL in both text and jsonb. See
src/aceh_rag/database/postgres_repository.py and the storage/backfill regression
tests/integration/test_postgres_repository.py::test_nul_free_provenance_survives_postgres_and_facts_backfill.
Chunking and indexing
The architecture targets roughly 512-token windows with overlap. The service uses a configurable chunk size and preserves character spans. Approved chunks get embeddings and lexical text. PostgreSQL stores HNSW vector index data and generated full-text search vectors.
Storage design
| Storage | Data |
|---|---|
| OBS / S3-compatible object storage | Raw uploads, batch corpus objects, future processed artifacts |
PostgreSQL documents | Stable document identity and source metadata |
PostgreSQL chunks | Text, embeddings, lexical text, OCR confidence, citations metadata |
| pgvector HNSW | Semantic retrieval over approved embeddings |
| PostgreSQL GIN FTS | Lexical retrieval over approved lexical text |
| Redis / queue state store | Job state in local/in-memory path, DCS target in production |
| Kafka / DMS | Async document-processing events in target architecture |
Idempotency and re-ingestion
The worker message carries content_sha256, and document identity uses source type, source version, and stable source key. For batch ingestion, the full object key is used as the source key so same-basename files in different folders persist as separate documents. Re-ingesting the same source replaces stale chunks transactionally.
Byte-identical content is deduplicated on its content checksum. Before parse/chunk/embed the service looks the incoming content_sha256 up and returns the existing document when it hits. Because concurrent workers can both miss that check-then-act lookup, an authoritative backstop makes the dedup deterministic: a partial functional unique index on documents over the extracted content_sha256 and sourceVersion (migration 0006_document_content_sha256_dedup) turns a racing duplicate insert into the same deduplicated outcome rather than a second document or a 500. The index is authored only on our AI-engine documents table, and a differing sourceVersion is a deliberate re-ingest that is allowed to coexist. See src/aceh_rag/postgres_repository.py.
Availability to AI services
A document becomes available to AI services only after at least one approved chunk is indexed. needs_ocr_review chunks persist for review but are excluded from semantic and lexical retrieval. Search and answer surfaces can use only approved indexed chunks; their exact request fields, citation shape, status behavior, SSE lifecycle, and examples are defined in the Query and Search API reference.
Operational states
| State | Meaning |
|---|---|
queued | Upload accepted and waiting for worker processing |
processing | Worker has started ingestion |
done | Processing completed and approved chunks were persisted |
needs_review | Human review required, usually OCR confidence related |
failed / permanently_failed | Processing failed and will not complete without operator action |
retrying | Worker will retry after a transient failure |
skipped | Duplicate or already-processed work skipped |