Ingestion and document APIs

This page is the frontend integration contract for document intake and ingestion status. All routes are root-mounted HTTP endpoints. Use the same origin or configured API base URL, for example https://api.example.gov/documents; the Docusaurus route for this page is /docs/api/ingestion-documents.

OpenAPI is available as GET /openapi.json in every environment. Swagger UI is available at GET /docs outside production. Redoc is not enabled.

Routes at a glance

Method	Route	Body	Returns	Use for
`POST`	`/documents/txt`	JSON	`TxtIngestResponse`	Small inline text ingestion without a job
`POST`	`/documents`	`multipart/form-data`	`UploadJobResponse`	User-selected local files
`POST`	`/ocr`	`multipart/form-data`	`OcrResponse`	OCR-only extraction of one file, without indexing
`POST`	`/ingestion/batch`	JSON	`UploadJobResponse`	Object-storage prefix or explicit object keys
`POST`	`/ingestion/url`	JSON	`UploadJobResponse`	Allow-listed document URLs
`GET`	`/ingestion/jobs/{job_id}`	none	`UploadJobResponse`	Poll an async upload/batch/URL job
`GET`	`/documents`	optional query	`UploadDocumentResponse[]`	Build lists, filters, dashboards, review queues
`GET`	`/documents/{document_id}`	none	`UploadDocumentResponse`	Inspect one uploaded document record
`DELETE`	`/documents/{document_id}`	none	`204 No Content`	Irreversibly remove an indexed document, its chunks, and its raw object when present

Delete is available for indexed documents through DELETE /documents/{document_id}. There are no cancellation or WebSocket ingestion endpoints. Upload progress after request acceptance is observed by polling GET /ingestion/jobs/{job_id}.

Authentication and frontend boundary

Every route on this page requires the configured API-key header. The default header name is X-API-Key.

X-API-Key: <server-injected-api-key>

A missing or incorrect key returns:

{ "detail": "Invalid or missing API key" }

Browser code must not embed a shared API key. Browser code also must not let users forge trusted identity headers such as X-User-Subject, X-User-Role, X-User-Unit, or X-User-Level. Put these APIs behind one of these patterns:

a same-origin backend-for-frontend that injects the API key server-side;
an API gateway that injects the shared key after authenticating the user;
a server-side admin/upload service.

There is no app-level CORS contract. Do not rely on cross-origin browser calls unless your gateway owns CORS and credential injection.

Shared types and limits

Source type enum

Upload, batch, and URL intake accept this source type enum:

type DocumentSourceType = "JDIH" | "OpenData" | "PPID" | "SatuData" | "upload";

If omitted, source_type defaults to "upload".

source_version is optional for upload, batch, and URL intake. If omitted there, the service default is "m0-upload". Inline text ingestion is different: POST /documents/txt has source_version in JSON with default "m0".

Supported extensions and MIME types

All file-oriented paths use the filename extension to select the document format. Multipart upload also validates the submitted MIME type.

Format in response	Extensions	Accepted multipart MIME types
`txt`	`.txt`	`text/plain`, `application/octet-stream`
`md`	`.md`	`text/markdown`, `text/plain`, `application/octet-stream`
`html`	`.html`	`text/html`, `application/xhtml+xml`
`csv`	`.csv`	`text/csv`, `application/csv`, `application/vnd.ms-excel`, `application/octet-stream`
`docx`	`.docx`	`application/vnd.openxmlformats-officedocument.wordprocessingml.document`
`xlsx`	`.xlsx`	`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
`pptx`	`.pptx`	`application/vnd.openxmlformats-officedocument.presentationml.presentation`
`pdf`	`.pdf`	`application/pdf`
`image`	`.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.webp`, `.bmp`	`image/png`, `image/jpeg`, `image/tiff`, `image/webp`, `image/bmp`

For batch and URL jobs, the initial content_type in the document response is application/octet-stream; the object key or URL extension still determines the format.

Born-digital PDFs that contain tables are parsed through a selective table-structure route so table content is preserved rather than flattened into unreadable runs of cells (#203). This affects the extracted text that is chunked and indexed; it does not change the request or response contract on this page.

Default limits

Limit	Default	Applies to
Per uploaded file	10 MiB	`POST /documents`; URL fetch stream cap uses the same per-file byte setting
Total multipart batch body	25 MiB	`POST /documents`
Files per multipart upload	20	`POST /documents`
URLs per request	20	`POST /ingestion/url` request schema
Archive entries	10,000	Archive-like documents before parse/OCR
Archive uncompressed bytes	100 MiB	Archive-like documents before parse/OCR
Archive compression ratio	100.0	Archive-like documents before parse/OCR
URL redirects	3	Worker fetch for URL ingestion
URL connect timeout	10 seconds	Worker fetch for URL ingestion
URL read timeout	30 seconds	Worker fetch for URL ingestion

POST /ingestion/batch does not enforce the multipart file-count or byte-body caps because it sends object references, not document bytes.

Job and document status model

Upload job response

POST /documents, POST /ingestion/batch, POST /ingestion/url, and GET /ingestion/jobs/{job_id} return this shape:

{
  "job_id": "job-1b2c3d4e...",
  "status": "queued",
  "documents": [
    {
      "id": "doc-1b2c3d4e...",
      "filename": "qanun-aceh.pdf",
      "title": "qanun-aceh",
      "content_type": "application/pdf",
      "format": "pdf",
      "content_sha256": "64-character lowercase sha256 hex",
      "object_ref": "s3://bucket/uploads/doc-.../qanun-aceh.pdf",
      "status": "queued",
      "detail": null,
      "stored_document_id": null,
      "chunk_count": 0
    }
  ],
  "created_at": "2026-07-23T10:15:30.123456+00:00",
  "updated_at": "2026-07-23T10:15:30.123456+00:00"
}

Document response fields

Field	Type	Meaning
`id`	string	Upload-tracking document ID. Use this with `GET /documents/{document_id}`.
`filename`	string	Sanitized basename. Path separators and unsafe filename characters are not preserved.
`title`	string	Filename stem, or filename when no stem is available.
`content_type`	string	Multipart MIME type, or `application/octet-stream` for batch/URL intake records.
`format`	string	One of `txt`, `md`, `html`, `csv`, `docx`, `xlsx`, `pptx`, `pdf`, `image`.
`content_sha256`	string	SHA-256 used for worker idempotency. For direct uploads it hashes file bytes; for batch and URL enqueue records it hashes the object key or URL seed until the worker reads bytes.
`object_ref`	string	Stored object reference, object key, or source URL depending on intake route. Treat it as display/debug metadata, not a download URL.
`status`	`UploadStatus`	Current document lifecycle state. See statuses below.
`detail`	string \| null	Human-readable failure or retry reason when available. Always show this for failed/review states.
`stored_document_id`	number \| null	Integer ID of the persisted searchable/reviewable document after worker ingestion creates it. Null before persistence or on enqueue-time skipped failures.
`chunk_count`	number	Number of chunks reported by the worker for this document. `0` is valid, especially for `queued`, failures, and some `needs_review` cases.

Upload statuses

type UploadStatus =
  | "queued"
  | "processing"
  | "retrying"
  | "done"
  | "needs_review"
  | "skipped"
  | "failed"
  | "permanently_failed";

Status	Terminal?	UI handling
`queued`	No	Accepted and waiting for the worker. Keep polling.
`processing`	No	Worker has started. Keep polling.
`retrying`	No	Transient error; worker will retry. Show `detail` if present and keep polling.
`done`	Yes	Processing completed. `stored_document_id` identifies the stored document and `chunk_count` reports chunks.
`needs_review`	Yes	Human review is required, usually because OCR/low-signal gates prevented normal indexing. Route to review; do not wait forever for `done`. Surface child documents with this status.
`skipped`	Yes	Terminal and benign — nothing was re-embedded. Two senses, both `skipped`: a byte-identical duplicate that was deduplicated (see Content deduplication), or a structured Open Data / Satu Data source that is tool-retrieved at query time rather than embedded. Show `detail` (for example `already done`); treat as success-without-new-indexing.
`failed`	Yes	Failed or skipped. Show `detail`. Batch unsupported-extension skips are reported this way.
`permanently_failed`	Yes	Worker-terminal failure after retry/dead-letter handling. Show `detail` and allow operator/user remediation.

Content deduplication

Ingestion is content-checksum deduplicated (#129, with an authoritative Postgres unique-index backstop for concurrent writers, #222). When a document whose content_sha256 and source_version already exist is re-submitted, the worker does not re-embed it; the child document reaches a terminal skipped status with detail such as already done, and the original stored document is left untouched. A byte-identical re-upload is therefore safe and idempotent rather than producing a duplicate corpus entry. Deduplication is by exact content bytes plus source_version: the same bytes ingested under a different source_version are treated as a distinct document.

Aggregate job status

A job status is calculated from child document statuses:

If every child is done, the job is done.
Otherwise, the first matching status in this precedence wins: permanently_failed, failed, needs_review, processing, retrying.
If none of those are present, the job remains queued.

Because aggregate status is intentionally conservative, always inspect and render documents[]. A job can include a mix of successful, failed, skipped, review-needed, and still-active documents. The aggregate status is a summary only: it can report failed, needs_review, or permanently_failed while lower-precedence siblings are still processing or retrying. Frontend completion logic must require every child document status to be terminal before exiting the polling loop, then branch from the child documents:

route needs_review children to human review;
show failed and permanently_failed children with their detail text;
show successful children with stored_document_id and chunk_count.

`POST /documents/txt`

Synchronous inline text ingestion. This route does not create an upload job and does not return upload-tracking document IDs.

Request JSON

{
  "title": "Qanun Aceh example",
  "text": "Full text to index...",
  "source_version": "m0"
}

Field	Required	Type	Validation/default
`title`	Yes	string	Minimum length 1
`text`	Yes	string	Minimum length 1; must contain indexable text
`source_version`	No	string	Minimum length 1; default `"m0"`

Response `200`

{
  "document_id": 123,
  "chunk_count": 4,
  "status": "indexed"
}

status is the stored document status for inline ingestion, not UploadStatus. On a normal successful text ingest it is indexed.

Errors

Status	Shape	When
`400`	`{ "detail": "..." }`	Empty or non-indexable document text
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`422`	FastAPI validation array	Missing fields, empty strings, wrong JSON types
`502`	`{ "detail": "..." }`	Model/embedding client failure

curl

curl -sS -X POST "$API_BASE/documents/txt" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Qanun Aceh example",
    "text": "Full text to index...",
    "source_version": "m0"
  }'

`POST /documents`

Multipart upload for one or more local files. The API stores bytes, creates one job, queues each file, and returns immediately with pollable IDs.

Form fields

Field	Required	Type	Validation/default
`files`	Yes	repeated file field	1-20 files; supported extension and multipart MIME type; each file ≤ 10 MiB; combined batch ≤ 25 MiB
`source_type`	No	`DocumentSourceType`	Default `"upload"`
`source_version`	No	string	Minimum length 1 when present; service default `"m0-upload"`

Use the same field name, files, for every uploaded file part.

Response `200`

Returns UploadJobResponse. Initial child document statuses are usually queued.

Errors

Status	Shape	When
`400`	`{ "detail": "..." }`	Too many files, batch too large, file too large, unsafe/missing filename, unsupported extension, unsupported MIME type, archive limit violation
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`422`	FastAPI validation array	Omitted `files` field, malformed multipart body, invalid `source_type`, empty `source_version`
`503`	`{ "detail": "Object storage is unavailable; the upload was not stored." }`	Object storage failure before enqueue completion

curl

curl -sS -X POST "$API_BASE/documents" \
  -H "X-API-Key: $API_KEY" \
  -F "source_type=upload" \
  -F "source_version=pilot-2026-07" \
  -F "files=@./qanun-aceh.pdf;type=application/pdf" \
  -F "files=@./lampiran.csv;type=text/csv"

`POST /ocr`

OCR-only extraction of a single uploaded PDF or image. This route runs the same parse-then-OCR-fallback used by ingestion, then returns the extracted text and its OCR confidence — but it stops before embedding or indexing, so nothing is added to the corpus. Use it to evaluate OCR quality on a document by itself, separate from retrieval and answer quality, or to preview what OCR would extract before deciding to ingest.

A born-digital PDF that already has a text layer is parsed without running OCR: ocr_performed is false and the confidence fields are null. A scanned PDF or an image is OCR'd and carries confidence scores.

Request

multipart/form-data with a single file part named file. Authentication and size limits match POST /documents.

Field	Required	Type	Validation
`file`	Yes	file	One file. Accepted content types are `application/pdf`, `image/png`, `image/jpeg`, `image/webp`. Each file ≤ 10 MiB (the same per-file byte limit as `POST /documents`).

Response `200`

{
  "format": "image",
  "text": "PEMERINTAH ACEH\nKEPUTUSAN ...",
  "ocr_performed": true,
  "model": "PaddleOCR-VL-0.9B",
  "mean_confidence": 0.94,
  "min_confidence": 0.71,
  "confidence_scores_comparable_with_prod": true,
  "confidence_threshold": 0.6,
  "review_required": false,
  "blocks": [
    {
      "text": "PEMERINTAH ACEH",
      "confidence": 0.98,
      "block_type": "title",
      "page_number": 1
    },
    {
      "text": "KEPUTUSAN ...",
      "confidence": 0.71,
      "block_type": "paragraph",
      "page_number": 1
    }
  ]
}

Field	Type	Meaning
`format`	string	Detected document format, for example `pdf` or `image`.
`text`	string	Full extracted text for the document.
`ocr_performed`	boolean	`true` when OCR ran; `false` when a born-digital text layer was used instead.
`model`	string \| null	OCR model identifier when OCR ran; `null` when `ocr_performed` is `false`.
`mean_confidence`	number \| null	Mean per-block OCR confidence; `null` when OCR did not run.
`min_confidence`	number \| null	Lowest per-block OCR confidence; `null` when OCR did not run.
`confidence_scores_comparable_with_prod`	boolean \| null	Whether these scores are comparable with the production OCR model version (OCR model-version drift makes old and new scores incomparable); `null` when OCR did not run.
`confidence_threshold`	number	The confidence gate threshold (ADR-0005). A score below it flags the extraction for human review.
`review_required`	boolean	`true` when the extraction is below the confidence gate and should route to human review before it is trusted.
`blocks`	block array	Per-block extraction. Each block has `text`, `confidence` (number, or `null` for a born-digital block parsed without OCR), `block_type` (for example `title`, `paragraph`, `table`), and `page_number` (integer or `null`).

This route only extracts and scores; it never writes to the corpus. To ingest a document, use POST /documents, POST /ingestion/batch, or POST /ingestion/url.

Errors

Status	Shape	When
`400`	`{ "detail": "..." }`	Unsupported file type (use PNG, JPEG, WEBP, or PDF), or a document from which no text could be extracted
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`413`	`{ "detail": "..." }`	File exceeds the per-file size limit
`502`	`{ "detail": "..." }`	The OCR backend is unavailable. The message names only the failed role (`OCR`) and never the internal model host, so a caller cannot enumerate private inference endpoints; the full diagnostic goes to the server log.

curl

curl -sS -X POST "$API_BASE/ocr" \
  -H "X-API-Key: $API_KEY" \
  -F "file=@./scan.pdf;type=application/pdf"

`POST /ingestion/batch`

Batch intake references objects that already exist in configured object storage. The request sends either a prefix to scan or an explicit manifest of keys. Document bytes do not ride the HTTP request.

Request JSON: prefix scan

{
  "prefix": "corpus/jdih/2026/",
  "source_type": "JDIH",
  "source_version": "jdih-2026-07"
}

Request JSON: explicit keys

{
  "keys": [
    "corpus/jdih/2026/qanun-1.pdf",
    "corpus/jdih/2026/qanun-2.docx"
  ],
  "source_type": "JDIH",
  "source_version": "jdih-2026-07"
}

Field	Required	Type	Validation/default
`prefix`	Required iff `keys` absent	string \| null	Minimum length 1 at JSON validation; whitespace-only prefixes are rejected by service validation; must match at least one object
`keys`	Required iff `prefix` absent	string[] \| null	Must not be empty; duplicate keys are deduplicated in first-seen order
`source_type`	No	`DocumentSourceType`	Default `"upload"`
`source_version`	No	string \| null	Minimum length 1 when present; service default `"m0-upload"`

Prefix-vs-keys validation

Provide exactly one of prefix or keys:

both omitted: validation error;
both present: validation error;
keys: []: validation error;
prefix: "": validation error;
prefix: " ": 400 with Prefix must not be empty;
prefix with no matching objects: 400 with No objects found under prefix: ....

Unsupported object extensions do not reject the whole batch. They are included in documents[] as failed records with detail like skipped: [filename] has an unsupported file extension, while supported objects are queued.

Response `200`

Returns UploadJobResponse. Batch child records use:

object_ref: the object key;
content_type: application/octet-stream;
content_sha256: a SHA-256 seed derived from the object key until the worker reads the bytes.

Errors

Status	Shape	When
`400`	`{ "detail": "..." }`	Whitespace prefix, no prefix matches, service-level batch validation failure
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`422`	FastAPI validation array	Invalid JSON shape, both/neither `prefix` and `keys`, empty keys, invalid `source_type`, empty `source_version`, extra fields
`503`	`{ "detail": "Object storage is unavailable; the batch was not enqueued." }`	Object storage list/read dependency unavailable during enqueue

curl

curl -sS -X POST "$API_BASE/ingestion/batch" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keys": ["corpus/jdih/qanun-1.pdf", "corpus/jdih/qanun-2.docx"],
    "source_type": "JDIH",
    "source_version": "jdih-2026-07"
  }'

`POST /ingestion/url`

URL intake enqueues one or more allow-listed document URLs. The API validates the URL list and creates a job; the worker fetches bytes later. No network fetch happens during the HTTP request.

Feature flag and allow-list

URL ingestion is disabled by default. After the request is authenticated and schema-valid, a disabled endpoint returns 403 before service URL validation or enqueue:

{
  "detail": "URL ingestion is disabled. Enable ACEH_RAG_URL_INGESTION__ENABLED and configure an allow-list to use this endpoint."
}

When enabled, the URL host must match the configured allow-list. Matching is case-insensitive. A plain entry matches exactly, for example jdih.acehprov.go.id. A leading-dot entry also matches the bare domain and subdomains, for example .go.id matches go.id and jdih.acehprov.go.id.

Only http and https schemes are accepted. The URL path basename must have a supported document extension. Query strings are ignored when deriving the filename.

At worker fetch time, every redirect hop is re-validated against scheme, allow-list, resolved IP backstops, redirect count, and stream size. Fetch-time failures affect that child document status; they do not change the already-created HTTP response.

Request JSON

{
  "urls": [
    "https://jdih.acehprov.go.id/dokumen/qanun-aceh-1-2026.pdf"
  ],
  "source_type": "JDIH",
  "source_version": "jdih-2026-07"
}

Field	Required	Type	Validation/default
`urls`	Yes	string[]	1-20 items by request schema; blank items are trimmed/dropped by service validation; duplicates are deduplicated in first-seen order
`source_type`	No	`DocumentSourceType`	Default `"upload"`
`source_version`	No	string \| null	Minimum length 1 when present; service default `"m0-upload"`

The request is all-or-nothing at enqueue time: one invalid URL rejects the whole request and no job is created.

Response `200`

Returns UploadJobResponse. URL child records use:

object_ref: the submitted URL;
content_type: application/octet-stream;
format: derived from the URL path extension;
content_sha256: a SHA-256 seed derived from the URL until the worker fetches bytes.

Errors

Status	Shape	When
`400`	`{ "detail": "..." }`	Empty/non-empty URL validation failure, unsupported scheme, missing or invalid host, off-allow-list host, no recognized extension, unsupported extension, more URLs than the configured service cap after dedupe
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`403`	`{ "detail": "URL ingestion is disabled..." }`	Authenticated, schema-valid request while the feature flag is off
`422`	FastAPI validation array	Missing `urls`, more than 20 submitted URL items, invalid `source_type`, empty `source_version`, extra fields

Fetch-time redirect, DNS, internal-IP/metadata-IP, size-limit, timeout, or transport failures appear later on the affected child document as failed or permanently_failed with detail populated.

curl

curl -sS -X POST "$API_BASE/ingestion/url" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://jdih.acehprov.go.id/dokumen/qanun-aceh-1-2026.pdf"],
    "source_type": "JDIH",
    "source_version": "jdih-2026-07"
  }'

`GET /ingestion/jobs/{job_id}`

Polls one upload, batch, or URL ingestion job.

Path parameters

Parameter	Type	Meaning
`job_id`	string	Job ID returned by `POST /documents`, `POST /ingestion/batch`, or `POST /ingestion/url`

Response `200`

Returns current UploadJobResponse. The server refreshes document state from worker attempts before responding.

Errors

Status	Shape	When
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`404`	`{ "detail": "Ingestion job not found" }`	Unknown job ID

Polling terminal handling

Use child statuses for the UI and polling loop control. The aggregate job status is only a summary; needs_review is terminal for an individual document.

const TERMINAL_UPLOAD_STATUSES = new Set([
  "done",
  "needs_review",
  "skipped",
  "failed",
  "permanently_failed",
]);

function isTerminalDocument(doc: UploadDocumentResponse): boolean {
  return TERMINAL_UPLOAD_STATUSES.has(doc.status);
}

function isTerminalJob(job: UploadJobResponse): boolean {
  return job.documents.every(isTerminalDocument);
}

function childFailures(job: UploadJobResponse): UploadDocumentResponse[] {
  return job.documents.filter((doc) =>
    doc.status === "failed" || doc.status === "permanently_failed"
  );
}

function reviewDocuments(job: UploadJobResponse): UploadDocumentResponse[] {
  return job.documents.filter((doc) => doc.status === "needs_review");
}

If polling times out in your UI, keep the last response visible. Do not convert a local polling timeout into a server failure.

`GET /documents`

Lists upload-tracking document records known to the upload/job service.

Query parameters

Parameter	Type	Required	Meaning
`status`	`UploadStatus`	No	When present, returns only documents with that status. Invalid values return `422`.

Examples:

GET /documents
GET /documents?status=needs_review
GET /documents?status=failed

Response `200`

[
  {
    "id": "doc-1b2c3d4e...",
    "filename": "scan.pdf",
    "title": "scan",
    "content_type": "application/pdf",
    "format": "pdf",
    "content_sha256": "64-character lowercase sha256 hex",
    "object_ref": "s3://bucket/uploads/doc-.../scan.pdf",
    "status": "needs_review",
    "detail": null,
    "stored_document_id": 123,
    "chunk_count": 0
  }
]

Use this route for upload dashboards, failure lists, and review queues. For human-review routing, filter status=needs_review and use stored_document_id when it is present.

Errors

Status	Shape	When
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`422`	FastAPI validation array	Invalid `status` query enum

`GET /documents/{document_id}`

Fetches one upload-tracking document record by its id field from UploadDocumentResponse.

Path parameters

Parameter	Type	Meaning
`document_id`	string	Upload-tracking document ID, usually shaped like `doc-...`

Response `200`

Returns one UploadDocumentResponse object.

Errors

Status	Shape	When
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`404`	`{ "detail": "Document not found" }`	Unknown document ID

`DELETE /documents/{document_id}`

Hard-deletes an indexed document and is irreversible. This route addresses the stored searchable document's UUID (documents.id), not the upload-tracking id returned by POST /documents (usually doc-...).

Authentication and authorization

The route requires the configured API-key header (the default is X-API-Key). In deployments with require_auth=true, it also requires a verified principal with the admin role; verified operator and user principals receive 403. A missing or malformed authenticated principal receives 401. With require_auth=false (the default in this codebase), the role is not enforced and the development/default principal path is used.

Deletion behavior

The service deletes every chunk associated with the document and the documents row in one database transaction, removing the document from both semantic and lexical retrieval. After that transaction commits, it deletes the raw uploaded object when one is recorded. A missing raw object is treated as a no-op; documents without a raw object still complete after their index rows are removed. The deletion is audited with the actor, document ID, and outcome, without document contents or credentials.

Response `204`

On full success, including a successful or unnecessary raw-object delete, the response has status 204 and no body. Repeating a delete after the document is gone returns 404.

Errors

Status	Shape	When
`401`	`{ "detail": "Invalid or missing API key" }`	Missing or wrong API key
`401`	`{ "detail": "Authenticated principal required" }`	No authenticated principal when `require_auth=true`
`401`	`{ "detail": "Invalid authenticated principal" }`	Malformed authenticated identity when `require_auth=true`, or a present subject that is not a UUID
`403`	`{ "detail": "admin role required" }`	Verified principal is not an `admin` when `require_auth=true`
`404`	`{ "detail": "Document not found" }`	Unknown, malformed, or already-deleted document ID; malformed IDs are intentionally indistinguishable from unknown IDs
`500`	`{ "detail": "Document removed from the index but its raw object could not be deleted" }`	Database cleanup committed, but raw-object deletion failed; the index is clean, the raw object is orphaned, and the failure is audited

Upload and poll from TypeScript

Run this from a trusted server-side BFF or gateway-adjacent service. Do not expose API_KEY to browser JavaScript.

type DocumentSourceType = "JDIH" | "OpenData" | "PPID" | "SatuData" | "upload";
type UploadStatus =
  | "queued"
  | "processing"
  | "retrying"
  | "done"
  | "needs_review"
  | "skipped"
  | "failed"
  | "permanently_failed";

type UploadDocumentResponse = {
  id: string;
  filename: string;
  title: string;
  content_type: string;
  format: "txt" | "md" | "html" | "csv" | "docx" | "xlsx" | "pptx" | "pdf" | "image" | string;
  content_sha256: string;
  object_ref: string;
  status: UploadStatus;
  detail: string | null;
  stored_document_id: number | null;
  chunk_count: number;
};

type UploadJobResponse = {
  job_id: string;
  status: UploadStatus;
  documents: UploadDocumentResponse[];
  created_at: string;
  updated_at: string;
};

const TERMINAL = new Set<UploadStatus>([
  "done",
  "needs_review",
  "skipped",
  "failed",
  "permanently_failed",
]);

async function apiFetch(path: string, init: RequestInit = {}) {
  const response = await fetch(`${process.env.API_BASE}${path}`, {
    ...init,
    headers: {
      "X-API-Key": process.env.API_KEY!,
      ...init.headers,
    },
  });

  if (!response.ok) {
    const body = await response.json().catch(() => undefined);
    throw new Error(
      `${response.status} ${response.statusText}: ${JSON.stringify(body)}`
    );
  }

  return response;
}

export async function uploadDocuments(files: File[]): Promise<UploadJobResponse> {
  const form = new FormData();
  form.set("source_type", "upload" satisfies DocumentSourceType);
  form.set("source_version", "frontend-2026-07");
  for (const file of files) form.append("files", file, file.name);

  const response = await apiFetch("/documents", {
    method: "POST",
    body: form,
  });
  return (await response.json()) as UploadJobResponse;
}

export async function createBatchIngestion(input:
  | { prefix: string; keys?: never; source_type?: DocumentSourceType; source_version?: string }
  | { keys: string[]; prefix?: never; source_type?: DocumentSourceType; source_version?: string }
): Promise<UploadJobResponse> {
  const response = await apiFetch("/ingestion/batch", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      ...input,
      source_type: input.source_type ?? "upload",
    }),
  });
  return (await response.json()) as UploadJobResponse;
}

export async function createUrlIngestion(input: {
  urls: string[];
  source_type?: DocumentSourceType;
  source_version?: string;
}): Promise<UploadJobResponse> {
  const response = await apiFetch("/ingestion/url", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      ...input,
      source_type: input.source_type ?? "upload",
    }),
  });
  return (await response.json()) as UploadJobResponse;
}

export async function pollIngestionJob(
  jobId: string,
  options: { intervalMs?: number; timeoutMs?: number } = {}
): Promise<UploadJobResponse> {
  const intervalMs = options.intervalMs ?? 1_000;
  const timeoutMs = options.timeoutMs ?? 120_000;
  const deadline = Date.now() + timeoutMs;
  let last: UploadJobResponse | undefined;

  while (Date.now() < deadline) {
    const response = await apiFetch(`/ingestion/jobs/${encodeURIComponent(jobId)}`);
    const job = (await response.json()) as UploadJobResponse;
    last = job;

    // Always surface child-level outcomes; aggregate status is summary only and can
    // be terminal while lower-precedence siblings are still active.
    const failed = job.documents.filter(
      (doc) => doc.status === "failed" || doc.status === "permanently_failed"
    );
    const needsReview = job.documents.filter((doc) => doc.status === "needs_review");
    if (failed.length > 0) console.warn("Ingestion child failures", failed);
    if (needsReview.length > 0) console.info("Documents need review", needsReview);

    if (job.documents.every((doc) => TERMINAL.has(doc.status))) return job;
    await new Promise((resolve) => setTimeout(resolve, intervalMs));
  }

  throw new Error(`Timed out waiting for ingestion job ${jobId}; last=${JSON.stringify(last)}`);
}

Validation and error behavior

Error shapes

Most non-validation errors use:

{ "detail": "human-readable message" }

FastAPI request validation errors use an array in detail:

{
  "detail": [
    {
      "loc": ["body", "urls"],
      "msg": "Field required",
      "type": "missing"
    }
  ]
}

Build UI validation from the request tables above, but still render server detail values because limits and allow-lists are deployment-configurable.

Client-side validation checklist

Require at least one file for multipart upload.
Check file extension and MIME type before upload.
Enforce visible defaults: 20 files, 10 MiB per file, 25 MiB total multipart batch.
For batch intake, require exactly one of prefix or keys; do not submit both.
For URL intake, require 1-20 URL strings, http or https, supported extension in the path, and an operator-approved domain list if your UI can display it.
Treat needs_review as a terminal success-with-action state, not as a failure and not as still processing.
Surface every child document whose status is failed, permanently_failed, or needs_review even when the aggregate job status is already terminal.

/docs/api/overview for base URL, OpenAPI, and route map conventions.
/docs/api/errors for shared HTTP error handling.

Routes at a glance​

Authentication and frontend boundary​

Shared types and limits​

Source type enum​

Supported extensions and MIME types​

Default limits​

Job and document status model​

Upload job response​

Document response fields​

Upload statuses​

Content deduplication​

Aggregate job status​

POST /documents/txt​

Request JSON​

Response 200​

Errors​

curl​

POST /documents​

Form fields​

Response 200​

Errors​

curl​

POST /ocr​

Request​

Response 200​

Errors​

curl​

POST /ingestion/batch​

Request JSON: prefix scan​

Request JSON: explicit keys​

Prefix-vs-keys validation​

Response 200​

Errors​

curl​

POST /ingestion/url​

Feature flag and allow-list​

Request JSON​

Response 200​

Errors​

curl​

GET /ingestion/jobs/{job_id}​

Path parameters​

Response 200​

Errors​

Polling terminal handling​

GET /documents​

Query parameters​

Response 200​

Errors​

GET /documents/{document_id}​

Path parameters​

Response 200​

Errors​

DELETE /documents/{document_id}​

Authentication and authorization​

Deletion behavior​

Response 204​

Errors​

Upload and poll from TypeScript​

Validation and error behavior​

Error shapes​

Client-side validation checklist​

Related API pages​

Routes at a glance

Authentication and frontend boundary

Shared types and limits

Source type enum

Supported extensions and MIME types

Default limits

Job and document status model

Upload job response

Document response fields

Upload statuses

Content deduplication

Aggregate job status

`POST /documents/txt`

Request JSON

Response `200`

Errors

curl

`POST /documents`

Form fields

Response `200`

Errors

curl

`POST /ocr`

Request

Response `200`

Errors

curl

`POST /ingestion/batch`

Request JSON: prefix scan

Request JSON: explicit keys

Prefix-vs-keys validation

Response `200`

Errors

curl

`POST /ingestion/url`

Feature flag and allow-list

Request JSON

Response `200`

Errors

curl

`GET /ingestion/jobs/{job_id}`

Path parameters

Response `200`

Errors

Polling terminal handling

`GET /documents`

Query parameters

Response `200`

Errors

`GET /documents/{document_id}`

Path parameters

Response `200`

Errors

`DELETE /documents/{document_id}`

Authentication and authorization

Deletion behavior

Response `204`

Errors

Upload and poll from TypeScript

Validation and error behavior

Error shapes

Client-side validation checklist

Related API pages