Sending to Kafka is a two-phase commit
The work isn't done at producer.send() — it's done when the receiver has processed and acknowledged the message. Until then two nodes are inconsistent, and that eventual-consistency window has to be designed on purpose, not discovered in production.
Problem
Teams think: message sent — job done. But the system becomes consistent only when the receiver has processed and acknowledged. Until that moment, two nodes are in a misaligned state — and this window has to be designed on purpose, not ignored.
Methodology
- 1. What the two phases actually are. Phase 1 — the sender commits the change and publishes the event. Phase 2 — the receiver processes and acknowledges. Between phases the system is inconsistent: the sender is done, the receiver isn't yet. This isn't a bug — it's the normal state of eventual consistency. You design for it explicitly.
- 2. The receiver is not a dumb relay. The receiver has its own business logic: it can reject the event (item out of stock, limit exceeded). If the sender doesn't handle rejection, it stays misaligned forever.
- 3. Two architectural choices — both need a business conversation. Fire-and-forget: the sender treats the task as done immediately. Await confirmation: the sender waits for the receiver's acknowledgment. Neither is "right" on its own — the choice depends on what the business will accept: a window of inconsistency or extra latency. Make it explicit before you commit to an architecture.
-
4. Infrastructure prerequisite: outbox.
Without the outbox pattern, phase 1 is already unreliable: writing to the DB and
publishing the event are two separate commits, and they can diverge.
# Phase 1: service A commits the change and sends with db.transaction(): order.status = "placed" order_repo.save(order) outbox.publish("orders.created", order) # ← A is done; system is inconsistent until phase 2 completes # Phase 2: service B processes asynchronously def handle_order_created(event): if not inventory.reserve(event.items): outbox.publish("orders.rejected", event.order_id) return fulfillment_repo.create(event) outbox.publish("orders.confirmed", event.order_id) # Phase 1 completes: A receives confirmation from B def handle_order_confirmed(event): order_repo.update(event.order_id, status="confirmed") # ← only now are both nodes consistent on the business fact def handle_order_rejected(event): order_repo.update(event.order_id, status="rejected") -
5. Three "acknowledgments" that get collapsed into one.
"Sent = done" collapses three different facts, and each is its own knob with its
own guarantee:
Level What it confirms Knob What it does NOT mean Broker ack Message written to the broker's log acks(0/1/all) +min.insync.replicasThat anyone has processed it Offset commit Consumer marked the message as read enable.auto.commit/ manualcommitSyncThat the business logic ran Business confirmation Receiver processed it and agrees on the business fact application event confirmed/rejected— (this is phase 2) acks=all+min.insync.replicas=2is the floor for publish durability: the ack fires only once the write is replicated to the ISR, otherwise you lose it when the leader dies. But durability isn't processing — a broker ack closes reliability for phase 1, not phase 2. -
6. Offset-commit ordering sets your delivery semantics.
Commit before processing — at-most-once (crash mid-flight → the message is
gone). Commit after processing — at-least-once (crash after processing but
before the commit → redelivery, which is why consumer idempotency is mandatory).
Exactly-once on the publish side —
enable.idempotence=true+ transactions (transactional.id), binding the topic write and the offset commit into one atomic commit; on the processing side you still need a dedup key, because Kafka's "exactly-once" is about its own topics, not the side effects in your DB.
For tracking state across multiple services — Correlation ID (a single identifier that passes through all phases). For managing compensating actions on rejection — the Saga pattern.
Artifact
Video: Why sending a message to Kafka is a two-phase commit (YouTube @IT-Head, 3,600 views)
Where it breaks
- Dual write without outbox — the DB write committed, the publish failed (or vice versa): inconsistency that surfaces under load, not in the demo. Mitigation: transactional outbox + relay (CDC or a poller) — one transaction for both the write and the "intent to publish."
- Offset committed before processing — silent message loss when you crash mid-flight. Mitigation: manual commit strictly after successful processing (at-least-once) + consumer idempotency; turn auto-commit off when side effects matter.
- Consumer idempotency is mandatory — under retries the two-phase nature of the protocol produces duplicates by design. Mitigation: a dedup key (business key / correlation id) in a processed store, upsert instead of insert.
- Business isn't ready for the inconsistency window — if the stakeholder expects immediate consistency but the system is eventual consistency, that's a requirements conflict to close before the architecture, not after an incident.
-
Receiver rejected — sender doesn't know —
without an explicit
handle_rejected, the system hangs in an intermediate state forever. Mitigation: a mandatory compensating path (Saga) + a timeout/alert on anything stuck in an intermediate status beyond SLA. What to monitor: consumer lag, the rejected share, the age of records in an unclosed intermediate status.
For whom and why
For teams building event-driven integrations who want to understand exactly where inconsistency arises — and how to design for it, not discover it in production.
Keep reading
About the author
Grigoriy Dobryakov — fractional / interim eng + AI leadership
Want to design event-driven consistency for your system?
Outbox, idempotency, Saga, Correlation ID — the mechanics of building event-driven systems that fail predictably, not randomly.
Discuss a fractional audit Email meOther breakdowns
An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.
Back to series →