Data Ingestion
Data ingestion prepares raw government documents for retrieval and grounded answering. It validates inputs, stores raw bytes, parses or OCRs content, extracts metadata, chunks text, embeds approved chunks, builds lexical text, persists rows, and exposes processing status.
:::info Authoritative API contract This page explains ingestion architecture and lifecycle decisions. For request and response fields, status codes, error bodies, curl commands, and TypeScript examples, use the Ingestion and Documents API reference. Architecture summaries below are not the complete wire contract. :::
Supported sources
The data model recognizes five source types: JDIH, OpenData, PPID,
SatuData, and upload.
| Source | Current handling |
|---|---|
| JDIH | Legal documents follow the parse → OCR fallback → chunk → embed → index path. |
| PPID | Public-information documents follow the same ingestion path, including optional table-structure extraction for table-heavy born-digital PDFs. |
upload | Manual or integration files use the upload and batch APIs. |
| OpenData | Contract-compatible structured events are skipped before parse/chunk/embed/index; query-time Open Data reads are the source of truth when enabled. |
| SatuData | Contract-compatible structured events are skipped before parse/chunk/embed/index; query-time Open Data reads are the source of truth when enabled. |
The structured-source skip is deliberate: embedding a portal snapshot would
create a stale competing copy. The worker records a terminal skipped outcome
for internal upload jobs and emits skipped/skipped for Kafka stage tracking.
Live partner Kafka handoff is currently blocked because the pinned producer's
doc.uploaded payload does not provide the aiDocumentId required by the
consumer; the contracts must be reconciled before this path is integrated.
Supported formats
Docling Slim is the born-digital parser boundary for PDF, DOCX, PPTX, XLSX, CSV, MD, and HTML. TXT is handled locally. Images and scanned documents route through OCR.
Born-digital PDFs get selective table preservation. Prose PDFs stay on the fast pypdfium2 path, but a cheap table-density detector re-parses a table-heavy PDF through a docling/TableFormer table-structure route so tables survive as markdown tables carrying a heading-path breadcrumb into section_path, instead of being flattened into unreadable text. The table route is markedly slower and pulls a heavy optional dependency, so it is gated behind the detector and falls back to pypdfium2 when the converter is not provisioned. See src/aceh_rag/document_parser.py.
Ingestion entry points
The table below names the architectural entry points and what each path is responsible for. It intentionally omits exact payloads, response bodies, status codes, and error shapes; those belong in the Ingestion and Documents API reference.
| Method | Endpoint or seam | Architecture responsibility |
|---|---|---|
| Manual multipart upload | POST /documents | Accept user or integration files, persist raw bytes, create status-tracked ingestion work, and schedule per-file processing |
| Inline text | POST /documents/txt | Index text supplied directly by a trusted caller without routing through OCR |
| Batch object-storage prefix | POST /ingestion/batch with prefix | Discover objects under a configured prefix and enqueue matched corpus objects |
| Batch manifest | POST /ingestion/batch with explicit keys | Enqueue selected object keys from the configured object store |
| API integration | Ingestion API through the FastAPI boundary | Let external integrations enter the same validation, storage, queueing, and worker path as frontend uploads |
| Scheduled ingestion | Scheduler calling API or batch connectors | Reuse the same batch/source APIs as other integrations |
| Streaming ingestion | Kafka/DMS worker processing | Support async source-specific adapters without bypassing validation and indexing rules |
End-to-end pipeline
For OCR documents, each below-threshold or lost page is excluded from indexing.
The document-level needs_review verdict uses the configured failed-page
fraction, so one weak cover page does not automatically quarantine an otherwise
usable scan; a lost page always forces review. GET /documents exposes
chunk_count, indexed_chunk_count, and parked_chunk_count so clients can
distinguish a partially retrievable review outcome from a document with no
indexed chunks.
Validation
Validation happens before storage, before parsing, and before indexing.
| Stage | Validation |
|---|---|
| Upload request | API key, supported extension/type, file size, batch size |
| Object storage | Backend errors mapped to domain errors; partial upload cleanup prevents orphans |
| Worker message | Required fields: document ID, content SHA-256, filename, source type, source version |
| Raw bytes | Archive-bomb limits run before parse/OCR |
| Parsed text | Empty documents rejected |
| Chunk text | Low-signal/gibberish content routed to review before embedding |
| OCR chunks | Threshold gate on a model-version-comparable confidence (ADR-0005) |
| Repository | Duplicate ordinals rejected; indexed chunks require embeddings; needs_ocr_review chunks are excluded from retrieval |
Metadata extraction and persistence
The ingestion model stores document-level and chunk-level metadata.
Document rows contain title, source type, format, source version, ingest timestamp, status, and metadata JSON. Chunk rows contain document ID, ordinal, text, embedding, lexical text, generated lexical_tsv, OCR confidence, chunk status, page number, section path, legal reference, character spans, and metadata JSON.
The current ingestion service records parser metadata, OCR confidence threshold, content hash, source key, OCR model, confidence comparability, and page/source span fields.
Chunking and indexing
The architecture targets roughly 512-token windows with overlap. The service uses a configurable chunk size and preserves character spans. Approved chunks get embeddings and lexical text. PostgreSQL stores HNSW vector index data and generated full-text search vectors. On write, chunk text, lexical text, and metadata are stripped of NUL (0x00) bytes — which some source documents carry and PostgreSQL text columns reject — so a stray NUL no longer fails the chunk insert. See src/aceh_rag/postgres_repository.py.
Storage design
| Storage | Data |
|---|---|
| OBS / S3-compatible object storage | Raw uploads, batch corpus objects, future processed artifacts |
PostgreSQL documents | Stable document identity and source metadata |
PostgreSQL chunks | Text, embeddings, lexical text, OCR confidence, citations metadata |
| pgvector HNSW | Semantic retrieval over approved embeddings |
| PostgreSQL GIN FTS | Lexical retrieval over approved lexical text |
| Redis / queue state store | Job state in local/in-memory path, DCS target in production |
| Kafka / DMS | Async document-processing events in target architecture |
Idempotency and re-ingestion
The worker message carries content_sha256, and document identity uses source type, source version, and stable source key. For batch ingestion, the full object key is used as the source key so same-basename files in different folders persist as separate documents. Re-ingesting the same source replaces stale chunks transactionally.
Byte-identical content is deduplicated on its content checksum. Before parse/chunk/embed the service looks the incoming content_sha256 up and returns the existing document when it hits. Because concurrent workers can both miss that check-then-act lookup, an authoritative backstop makes the dedup deterministic: a partial functional unique index on documents over the extracted content_sha256 and sourceVersion (migration 0006_document_content_sha256_dedup) turns a racing duplicate insert into the same deduplicated outcome rather than a second document or a 500. The index is authored only on our AI-engine documents table, and a differing sourceVersion is a deliberate re-ingest that is allowed to coexist. See src/aceh_rag/postgres_repository.py.
Availability to AI services
A document becomes available to AI services only after at least one approved chunk is indexed. needs_ocr_review chunks persist for review but are excluded from semantic and lexical retrieval. Search and answer surfaces can use only approved indexed chunks; their exact request fields, citation shape, status behavior, SSE lifecycle, and examples are defined in the Query and Search API reference.
Operational states
| State | Meaning |
|---|---|
queued | Upload accepted and waiting for worker processing |
processing | Worker has started ingestion |
done | Processing completed and approved chunks were persisted |
needs_review | Human review required, usually OCR confidence related |
failed / permanently_failed | Processing failed and will not complete without operator action |
retrying | Worker will retry after a transient failure |
skipped | Duplicate or already-processed work skipped |