Sending to Kafka Is a Two-Phase Commit | Grigoriy Dobryakov

Grigoriy Dobryakov

Howto · breakdown

Breakdown 09 YouTube · 3,600 views

Sending to Kafka is a two-phase commit

The work isn't done at producer.send() — it's done when the receiver has processed and acknowledged the message. Until then two nodes are inconsistent, and that eventual-consistency window has to be designed on purpose, not discovered in production.

CTO Architect Tech Lead Head of AI

Problem

Teams think: message sent — job done. But the system becomes consistent only when the receiver has processed and acknowledged. Until that moment, two nodes are in a misaligned state — and this window has to be designed on purpose, not ignored.

Methodology

  1. 1. What the two phases actually are. Phase 1 — the sender commits the change and publishes the event. Phase 2 — the receiver processes and acknowledges. Between phases the system is inconsistent: the sender is done, the receiver isn't yet. This isn't a bug — it's the normal state of eventual consistency. You design for it explicitly.
  2. 2. The receiver is not a dumb relay. The receiver has its own business logic: it can reject the event (item out of stock, limit exceeded). If the sender doesn't handle rejection, it stays misaligned forever.
  3. 3. Two architectural choices — both need a business conversation. Fire-and-forget: the sender treats the task as done immediately. Await confirmation: the sender waits for the receiver's acknowledgment. Neither is "right" on its own — the choice depends on what the business will accept: a window of inconsistency or extra latency. Make it explicit before you commit to an architecture.
  4. 4. Infrastructure prerequisite: outbox. Without the outbox pattern, phase 1 is already unreliable: writing to the DB and publishing the event are two separate commits, and they can diverge.
    # Phase 1: service A commits the change and sends
    with db.transaction():
        order.status = "placed"
        order_repo.save(order)
        outbox.publish("orders.created", order)
    # ← A is done; system is inconsistent until phase 2 completes
    
    # Phase 2: service B processes asynchronously
    def handle_order_created(event):
        if not inventory.reserve(event.items):
            outbox.publish("orders.rejected", event.order_id)
            return
        fulfillment_repo.create(event)
        outbox.publish("orders.confirmed", event.order_id)
    
    # Phase 1 completes: A receives confirmation from B
    def handle_order_confirmed(event):
        order_repo.update(event.order_id, status="confirmed")
    # ← only now are both nodes consistent on the business fact
    
    def handle_order_rejected(event):
        order_repo.update(event.order_id, status="rejected")
  5. 5. Three "acknowledgments" that get collapsed into one. "Sent = done" collapses three different facts, and each is its own knob with its own guarantee:
    Level What it confirms Knob What it does NOT mean
    Broker ack Message written to the broker's log acks (0/1/all) + min.insync.replicas That anyone has processed it
    Offset commit Consumer marked the message as read enable.auto.commit / manual commitSync That the business logic ran
    Business confirmation Receiver processed it and agrees on the business fact application event confirmed/rejected — (this is phase 2)

    acks=all + min.insync.replicas=2 is the floor for publish durability: the ack fires only once the write is replicated to the ISR, otherwise you lose it when the leader dies. But durability isn't processing — a broker ack closes reliability for phase 1, not phase 2.

  6. 6. Offset-commit ordering sets your delivery semantics. Commit before processing — at-most-once (crash mid-flight → the message is gone). Commit after processing — at-least-once (crash after processing but before the commit → redelivery, which is why consumer idempotency is mandatory). Exactly-once on the publish side — enable.idempotence=true + transactions (transactional.id), binding the topic write and the offset commit into one atomic commit; on the processing side you still need a dedup key, because Kafka's "exactly-once" is about its own topics, not the side effects in your DB.

For tracking state across multiple services — Correlation ID (a single identifier that passes through all phases). For managing compensating actions on rejection — the Saga pattern.

Artifact

Video: Why sending a message to Kafka is a two-phase commit (YouTube @IT-Head, 3,600 views)

Where it breaks

For whom and why

For teams building event-driven integrations who want to understand exactly where inconsistency arises — and how to design for it, not discover it in production.

About the author

Grigoriy Dobryakov — fractional / interim eng + AI leadership

Want to design event-driven consistency for your system?

Outbox, idempotency, Saga, Correlation ID — the mechanics of building event-driven systems that fail predictably, not randomly.

Discuss a fractional audit Email me

Other breakdowns

An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.

Back to series →