Partner event API

This page is the self-contained contract for the partner backend integration at /docs/api/partner-events.

The consumer is a server-side partner backend that owns crawlers, a DMS, object storage keys, and Kafka job tracking. It is not a browser API. Do not send these events from browser code.

Transport and ownership

Aceh RAG consumes document-upload events from Kafka, reads the referenced bytes from configured object storage, runs the existing parse, OCR, chunk, embed, and index pipeline, then emits stage-status events back to Kafka for partner job tracking.

Direction	Default topic or map	Owner produces	Owner consumes	Purpose
Partner to Aceh RAG	doc.uploaded	Partner backend	Aceh RAG Kafka worker	Tell Aceh RAG that bytes are available under storageKey.
Aceh RAG to partner	kafka.stage_topics	Aceh RAG Kafka worker	Partner backend	Report per-document stage progress and terminal outcomes on the destination selected by stage.

Kafka is configuration-gated and defaults off for local, test, and CI processes. A staging or production deployment must explicitly enable and run the dedicated Kafka worker process. Topic names, broker list, and consumer group are configuration values, so confirm the active values for the target environment before connecting.

Setting	Default	Required deployment confirmation
kafka.enabled	false	Confirm it is enabled where the partner integration should run.
kafka.bootstrap_servers	localhost:9092	Confirm the real broker list.
kafka.doc_uploaded_topic	doc.uploaded	Confirm the input topic name.
kafka.stage_topics	Complete stage-to-topic map	Confirm every route and provision every destination.
kafka.consumer_group	aceh-rag-ingestion	Confirm the consumer group assigned to Aceh RAG.
kafka.default_source_version	m2-kafka	Confirm the fallback version stamped when sourceVersion is omitted.
kafka.max_retries	3	Confirm retry budget for transient fetch and pipeline failures.
kafka.retry_backoff_seconds	2.0	Confirm the base retry delay.

This contract does not specify broker security settings. Use the platform team's deployed Kafka connection profile.

Default stage-topic map

kafka.stage_topics is a complete mapping: worker configuration fails before startup if a route is missing, blank, or uses an unsupported key. The partner owns the actual DMS values and may override these defaults without changing the publisher.

Event condition	Wire stage	Default destination
Source bytes fetched	fetched	doc.extracted
Extraction / parse complete	parsed	doc.extracted
OCR begins (reserved; not emitted by the current pipeline)	ocr_pending	doc.ocr.pending
OCR complete	ocr	doc.ocr.done
Chunks produced	chunked	doc.chunked
Embeddings stored	embedded	doc.embedded
Index terminal success or review	indexed	doc.indexed
Terminal error	failed	doc.indexed
Terminal structured-source or content-duplicate no-op	skipped	doc.indexed

The partner must create the six agreed DMS destinations and grant the worker producer permission before deployment; broker auto-creation is not assumed.

doc.uploaded input event

Publish one JSON object per document to the configured input topic. The payload fields use camelCase.

Field table

Field	Required?	Type	Validation, default, and meaning
aiDocumentId	Yes	string	Must be nonblank. Stable partner document identity. Used for locking, idempotency, stage-event linkage, and the indexed document source_key.
storageKey	Yes	string	Must be nonblank. Object-storage key that Aceh RAG reads to fetch the document bytes.
sourceType	No	string enum	Must be one of JDIH, OpenData, PPID, SatuData, upload when present. Defaults to upload.
format	No	string	Optional nonblank format hint. See format values. If omitted, Aceh RAG derives the format from the storageKey basename extension.
sourceSystem	No	string	Optional nonblank source system label, for example jdih-portal or ppid-crawler. Persisted as source metadata when present.
collectionId	No	string	Optional nonblank partner collection identifier. Persisted as source metadata when present.
skpaUnit	No	string	Optional nonblank SKPA or organization unit. Persisted as source metadata when present.
classification	No	string	Optional nonblank classification label. Persisted as source metadata when present.
category	No	string	Optional nonblank partner category label. Persisted as source metadata when present.
uploadedBy	No	string	Optional nonblank actor or service account identifier. Used as trace user when present and persisted as source metadata.
sourceVersion	No	string	Optional nonblank version for this source document. If omitted, Aceh RAG uses the configured kafka.default_source_version, default m2-kafka.

Every optional field is optional, but if the field is present its value must be a nonblank string. null, an empty string, whitespace-only strings, arrays, objects, booleans, and numbers are invalid for these string fields. Unknown top-level fields are ignored by the current consumer and are not persisted as source metadata. Do not put required partner job state only in unknown fields.

Source type values

type DocumentSourceType = "JDIH" | "OpenData" | "PPID" | "SatuData" | "upload";

sourceType is a strict enum. Any other value makes the event malformed and produces a failed stage event.

Format values

Supported document formats are:

type DocumentFormat = "txt" | "md" | "html" | "csv" | "docx" | "xlsx" | "pptx" | "pdf" | "image";

format is a hint, not the document bytes. The current Kafka filename derivation uses the hint to force these extensions when storageKey is extensionless:

format hint	Derived extension
pdf	.pdf
docx	.docx
xlsx	.xlsx
csv	.csv
html	.html
image	.png

For txt, md, and pptx, or when format is omitted or unrecognized, make sure the storageKey basename already has the parseable extension, such as records/qanun.txt, records/page.md, or records/slides.pptx. If Aceh RAG cannot derive a parseable filename for the bytes, the pipeline fails and emits failed with status: "error".

Minimal event

{
  "aiDocumentId": "ai-doc-2026-0001",
  "storageKey": "partner-uploads/2026/qanun-aceh-0001.pdf"
}

This is valid. Aceh RAG applies sourceType: "upload" and the configured default sourceVersion.

Full event example

{
  "aiDocumentId": "jdih-aceh-qanun-2026-001",
  "storageKey": "jdih/2026/qanun-aceh-001",
  "sourceType": "JDIH",
  "format": "pdf",
  "sourceSystem": "jdih-portal",
  "collectionId": "jdih-aceh",
  "skpaUnit": "dinas-keuangan",
  "classification": "public",
  "category": "qanun",
  "uploadedBy": "crawler-jdih",
  "sourceVersion": "2026-07-23T10:15:30Z"
}

Because format is pdf, the extensionless storageKey basename is treated as a PDF filename for parser dispatch.

Input validation and failure behavior

Malformed messages are not silently dropped. Aceh RAG emits one failed stage event where possible, logs the rejection, and treats the message as terminal for offset handling.

Failure	Retry?	Stage events	Partner interpretation
Payload is not valid JSON	No	failed/error with aiDocumentId: "unknown" unless it can be salvaged	Mark the job failed if it can be matched. Otherwise send to an operator queue because the document identity is unknown.
Payload is JSON but not an object	No	failed/error	Same as malformed JSON.
Missing or blank aiDocumentId	No	failed/error with aiDocumentId: "unknown"	The partner cannot correlate automatically. Inspect the producer logs.
Missing or blank storageKey	No	failed/error linked to aiDocumentId when present	Fix and republish the event.
Unsupported sourceType	No	failed/error linked to aiDocumentId when present	Fix the enum value and republish.
Optional string field is present but blank or wrong type	No	failed/error linked to aiDocumentId when present	Remove the field or send a nonblank string, then republish.
Object-storage fetch fails after retries	Yes, bounded internally	failed/error	Terminal for this delivery. The partner may republish after fixing object availability.
Parse, OCR, chunk, embed, or index pipeline fails after retries	Yes, bounded internally	Usually fetched/ok, then failed/error	Terminal for this delivery. The partner may republish after fixing the cause.

Permanent malformed messages fail fast. Transient object-storage and pipeline errors are retried with exponential backoff using the configured retry settings. Retry attempts do not produce separate retry stage events.

Document identity, versioning, and idempotency

Use aiDocumentId as the stable document identity across systems.

Aceh RAG uses two related identities:

1. In-process duplicate detection uses (aiDocumentId, effectiveSourceVersion), where effectiveSourceVersion is the event sourceVersion or the configured default when omitted.
2. Stored document upsert uses source_key = aiDocumentId, which prevents duplicate stored rows for redelivery of the same document identity.

Partner rules:

• Republish the same aiDocumentId with the same sourceVersion only for redelivery or retry of the same version.
• Publish the same aiDocumentId with a new sourceVersion when the source document content or authoritative metadata has genuinely changed.
• Track jobs by aiDocumentId and, if your DMS versions documents, by sourceVersion too.
• If sourceVersion is omitted, all omitted-version deliveries use the deployment's configured default version for duplicate detection. Use explicit versions if partner-side version history matters.

When Aceh RAG sees a duplicate (aiDocumentId, effectiveSourceVersion) in the same worker process, it skips reprocessing and emits no new stage events. A cold-start redelivery can still run the pipeline again, but the stored document identity is upserted by aiDocumentId rather than creating duplicate rows.

StageStatusEvent output event

Aceh RAG publishes one JSON object per stage update to the configured destination selected by its stage. The Kafka message key is aiDocumentId, so events for the same document stay partition-ordered within a destination when the broker uses key partitioning. Kafka provides no total ordering across the separate stage topics.

JSON schema

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "StageStatusEvent",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "aiDocumentId",
    "stage",
    "status",
    "ocrConfidence",
    "chunkCount",
    "error",
    "at"
  ],
  "properties": {
    "aiDocumentId": { "type": "string", "minLength": 1 },
    "stage": {
      "type": "string",
      "enum": ["fetched", "parsed", "ocr", "chunked", "embedded", "indexed", "failed", "skipped"]
    },
    "status": {
      "type": "string",
      "enum": ["ok", "error", "needs_review", "skipped"]
    },
    "ocrConfidence": { "type": ["number", "null"] },
    "chunkCount": { "type": ["integer", "null"], "minimum": 0 },
    "error": { "type": ["string", "null"] },
    "at": { "type": "string", "format": "date-time" }
  }
}

The shape is stable. ocrConfidence, chunkCount, and error are always present and are null when not applicable.

Field meanings

Field	Type	Meaning
aiDocumentId	string	The partner document identity from doc.uploaded, or unknown when a malformed payload cannot provide one.
stage	enum	Current lifecycle stage. See stages.
status	enum	Stage outcome. See status meanings.
ocrConfidence	number | null	Lowest per-chunk OCR confidence for the document when an OCR stage is emitted. Null for non-OCR stages.
chunkCount	integer | null	For chunked, total chunks. For embedded and indexed, indexed chunk count. Null when not applicable.
error	string | null	Human-readable failure detail on failed/error. Null otherwise.
at	string	Producer timestamp in ISO 8601 format with timezone.

Stages

type Stage = "fetched" | "parsed" | "ocr" | "chunked" | "embedded" | "indexed" | "failed" | "skipped";

Stage	Meaning
fetched	Object bytes were read from storageKey.
parsed	The pipeline parsed the document content.
ocr	OCR ran and produced OCR confidence data. Born-digital documents usually skip this stage.
chunked	The parsed content was split into chunks. chunkCount is the total chunk count.
embedded	Chunks were passed through embedding and index preparation. chunkCount is the indexed chunk count.
indexed	Terminal successful or review-needed indexing outcome.
failed	Terminal failure pseudo-stage. It is used when validation, fetch, or pipeline handling cannot continue.
skipped	Terminal structured-source or content-duplicate no-op.

Status meanings

type StageStatus = "ok" | "error" | "needs_review" | "skipped";

Status	Terminal?	Meaning
ok	Only terminal on indexed	The stage completed. indexed/ok means the document has indexed chunks and the partner can mark that document version complete.
error	Yes	The event failed permanently for this delivery. The stage is failed. Show or store error, then fix and republish if the document should be retried.
needs_review	Yes	OCR-gated terminal outcome. Aceh RAG finished the pipeline, but the document produced zero indexed chunks and needs human review. This is emitted as indexed/needs_review, not indexed/ok. Do not mark the job complete.
skipped	Yes	Terminal no-op. This is emitted as skipped/skipped with chunkCount: 0 for a structured source or a content duplicate.

Ordered lifecycle

Stage events for one document are emitted in lifecycle order. fetched is emitted immediately after object bytes are fetched. The remaining success stages are emitted in order after the reused ingestion pipeline returns its result. Because those events cross topics, consumers must apply their existing stage/state rules rather than assume Kafka gives a total cross-topic order.

Born-digital success

fetched/ok
parsed/ok
chunked/ok
embedded/ok
indexed/ok

OCR success

fetched/ok
parsed/ok
ocr/ok
chunked/ok
embedded/ok
indexed/ok

OCR terminal review

fetched/ok
parsed/ok
ocr/ok
chunked/ok
embedded/ok
indexed/needs_review

indexed/needs_review is terminal for this delivery, but it is not a successful completion for partner job tracking. Route the document to human review.

Fetch failure

failed/error

No fetched/ok event is emitted when bytes cannot be fetched.

Pipeline failure after fetch

fetched/ok
failed/error

The failure may occur during parse, OCR, chunk, embed, or index work. The current output uses the terminal failed stage rather than a stage-specific error event.

Retry, deferred, skipped, and publish-error semantics

These states affect partner job tracking even when they are not all visible as output events.

Consumer outcome	Stage event?	Offset handling	Partner tracking rule
indexed	Yes, ends with indexed/ok	Committed	Mark the document version complete.
needs_review	Yes, ends with indexed/needs_review	Committed	Mark terminal review-needed, not complete.
error	Yes, usually failed/error	Committed	Mark failed for this delivery. Republish only after fixing the cause.
skipped	skipped/skipped for structured-source or content-duplicate no-ops; no new event for an already-processed redelivery	Committed	Treat a skipped/skipped event as terminal. For a redelivery already skipped by the in-process dedup store, retain the previous delivery's status.
deferred	No success stage event	Not committed for that record immediately	Another worker holds the per-document lock. Keep the job pending, reconcile it as a missing-terminal-status document, and explicitly re-drive it according to the agreed operational policy.

deferred is an internal, non-terminal outcome for lock contention. It emits no success stages and skips the contended record's immediate commit. Current consumer processing does not provide a same-partition redelivery guarantee: if a later record from the same partition is committed, that commit may advance the partition offset past the deferred record. Partner systems must reconcile documents that do not reach indexed/ok, indexed/needs_review, or failed/error and re-drive them explicitly under the agreed operational policy.

If publishing a stage-status event itself fails, Aceh RAG logs and continues processing the document. The document can still be persisted and marked processed even if one or more status events are missed. Partner consumers should make their stage updates idempotent and should have an operator reconciliation path for documents that were produced but have no recent status event.

Producer checklist for doc.uploaded

Before producing events:

1. Confirm the active input topic, complete stage-topic map, consumer group, broker list, and Kafka worker deployment with the platform team.
2. Ensure the object bytes exist at storageKey before publishing the event.
3. Use a stable nonblank aiDocumentId. Do not generate a new ID for a retry of the same source document.
4. Send sourceVersion when partner-side DMS versions matter. Bump it only for a real new source version.
5. Send sourceType only from JDIH, OpenData, PPID, SatuData, or upload.
6. Send a supported format hint or include a parseable extension in the storageKey basename.
7. Omit unknown optional fields rather than sending null or empty strings.
8. Keep the full original event in partner logs so a malformed or failed delivery can be corrected and republished.

Consumer checklist for StageStatusEvent

When consuming status events:

1. Key partner job state by aiDocumentId, and by sourceVersion too if your DMS tracks versions. The status event does not repeat sourceVersion, so retain it from the produced input event.
2. Treat all fields in the schema as required. Store nullable fields as nullable, not missing.
3. Apply stage updates idempotently. Duplicate status events for the same document and stage should not corrupt job state.
4. Preserve ordering within each destination topic, but tolerate missing intermediate stages and do not assume a total order across stage topics because stage publishing is best-effort.
5. Mark indexed/ok complete.
6. Mark indexed/needs_review as terminal review-needed and route to human review.
7. Mark failed/error as failed and surface error to operators.
8. Mark skipped/skipped as the terminal no-op status. Do not wait for a new status event after an already-processed duplicate redelivery; the previous delivery's status is authoritative for that version.
9. Reconcile pending jobs that have no terminal status event and explicitly re-drive them according to the agreed operational policy; do not rely on deferred lock contention to redeliver the original Kafka record.