Data Ingestion
Data ingestion prepares raw government documents for retrieval and grounded answering. It validates inputs, stores raw bytes, parses or OCRs content, extracts metadata, chunks text, embeds approved chunks, builds lexical text, persists rows, and exposes processing status.
Supported sources
The data model recognizes five source types: JDIH, OpenData, PPID, SatuData, and upload. Uploads use the ingestion service directly, and source connectors for JDIH, OpenData, PPID, and SatuData feed the same service so documents follow one validation, parsing, OCR, chunking, and indexing path.
| Source | Content profile | Ingestion status |
|---|---|---|
| JDIH | Laws, regulations, legal opinions, statute references | Source connector |
| OpenData | Structured/tabular public datasets | Source connector and uploaded/batch CSV/XLSX parsing |
| PPID | Public-information documents | Source connector |
| SatuData | Standardized government datasets | Source connector |
| Upload | Manual or batch files from users/integrations | Upload and batch APIs |
Supported formats
Docling Slim is the born-digital parser boundary for PDF, DOCX, PPTX, XLSX, CSV, MD, and HTML. TXT is handled locally. Images and scanned documents route through OCR.
Ingestion methods
| Method | Endpoint or seam | Current behavior |
|---|---|---|
| Manual multipart upload | POST /documents | Stores files, creates a job, queues each file, returns job and document IDs |
| Inline text | POST /documents/txt | Creates one document directly from request JSON and indexes it |
| Batch object-storage prefix | POST /ingestion/batch with prefix | Lists keys under a prefix and enqueues one job for matched objects |
| Batch manifest | POST /ingestion/batch with explicit keys | Enqueues selected object keys |
| API integration | Any endpoint via APIG/FastAPI | External clients call the same API, not a side path |
| Scheduled ingestion | Scheduler calling API/batch connector | Calls the same batch/source APIs as other integrations |
| Streaming ingestion | Kafka/DMS worker processing | Async queue processing supports source-specific adapters |
End-to-end pipeline
Validation
Validation happens before storage, before parsing, and before indexing.
| Stage | Validation |
|---|---|
| Upload request | API key, supported extension/type, file size, batch size |
| Object storage | Backend errors mapped to domain errors; partial upload cleanup prevents orphans |
| Worker message | Required fields: document ID, content SHA-256, filename, source type, source version |
| Raw bytes | Archive-bomb limits run before parse/OCR |
| Parsed text | Empty documents rejected |
| Chunk text | Low-signal/gibberish content routed to review before embedding |
| OCR chunks | Confidence threshold and confidence-comparability gate |
| Repository | Duplicate ordinals rejected; indexed chunks require embeddings; needs_ocr_review chunks are excluded from retrieval |
Metadata extraction and persistence
The ingestion model stores document-level and chunk-level metadata.
Document rows contain title, source type, format, source version, ingest timestamp, status, and metadata JSON. Chunk rows contain document ID, ordinal, text, embedding, lexical text, generated lexical_tsv, OCR confidence, chunk status, page number, section path, legal reference, character spans, and metadata JSON.
The current ingestion service records parser metadata, OCR confidence threshold, content hash, source key, OCR model, confidence comparability, and page/source span fields.
Chunking and indexing
The architecture targets roughly 512-token windows with overlap. The service uses a configurable chunk size and preserves character spans. Approved chunks get embeddings and lexical text. PostgreSQL stores HNSW vector index data and generated full-text search vectors.
Storage design
| Storage | Data |
|---|---|
| OBS / S3-compatible object storage | Raw uploads, batch corpus objects, future processed artifacts |
PostgreSQL documents | Stable document identity and source metadata |
PostgreSQL chunks | Text, embeddings, lexical text, OCR confidence, citations metadata |
| pgvector HNSW | Semantic retrieval over approved embeddings |
| PostgreSQL GIN FTS | Lexical retrieval over approved lexical text |
| Redis / queue state store | Job state in local/in-memory path, DCS target in production |
| Kafka / DMS | Async document-processing events in target architecture |
Idempotency and re-ingestion
The worker message carries content_sha256, and document identity uses source type, source version, and stable source key. For batch ingestion, the full object key is used as the source key so same-basename files in different folders persist as separate documents. Re-ingesting the same source replaces stale chunks transactionally.
Availability to AI services
A document becomes available to AI services only after at least one approved chunk is indexed. needs_ocr_review chunks persist for review but are excluded from semantic and lexical retrieval. /search can retrieve ranked chunks, and /query can synthesize a cited answer only from retrieved chunks.
Operational states
| State | Meaning |
|---|---|
queued | Upload accepted and waiting for worker processing |
processing | Worker has started ingestion |
done | Processing completed and approved chunks were persisted |
needs_review | Human review required, usually OCR confidence related |
failed / permanently_failed | Processing failed and will not complete without operator action |
retrying | Worker will retry after a transient failure |
skipped | Duplicate or already-processed work skipped |