Transactional Messaging

Context

A service rarely acts alone. When the Order service confirms an order, it does two things: it writes a row to its own database, and it tells the rest of the system that the order now exists — usually by publishing a message or event to a broker so the Inventory, Payment, and Notification services can react. Sagas, event sourcing, and event-driven read models all depend on this second step happening reliably.

So almost every interesting microservice does the same pair of actions on the same code path: commit a local change, then publish a message about it.

Problem

The trouble is that the database and the message broker are two independent systems with two independent transactions. There is no shared commit. That leaves a gap, and whatever order you choose, a crash in the gap corrupts the system.

If you publish first and then write to the database, a crash after publishing leaves a message claiming something happened that never did — a ghost event. If you write to the database first and then publish, a crash after the commit but before publishing leaves a change that the rest of the world never hears about — a lost event. You cannot wrap both in one transaction, because the broker is not enrolled in your database transaction (and two-phase commit across them is slow, fragile, and often unsupported).

This is the dual-write problem: two writes that must both happen or neither, across two systems that cannot share a transaction.

sequenceDiagram
  participant S as Service
  participant DB as Database
  participant B as Message Broker
  S->>DB: BEGIN, UPDATE order, COMMIT
  Note over S,B: crash happens HERE
  S--xB: publish OrderConfirmed (never sent)
  Note over DB,B: DB says confirmed, broker never heard — lost event

The dual-write hazard — a crash between the commit and the publish loses the event

Solution

Every pattern in this module follows the same insight: stop trying to write to two systems atomically. Instead, make the message part of the same local database transaction that records the business change, and move the job of getting it to the broker into a separate, retryable step. The database commit becomes the single source of truth, and message delivery becomes a consequence of it rather than a second, racing write.

The four patterns build on each other:

• Transactional Outbox — write the outgoing message into an outbox table in the same local transaction as the business change. Now both succeed or both roll back. A separate relay later delivers the saved messages to the broker.
• Transaction Log Tailing — implement the relay by reading the database’s own commit log (change data capture), so outbox inserts flow to the broker with no polling and almost no added latency.
• Polling Publisher — implement the relay by periodically querying the outbox table for unsent rows and publishing them; simpler to operate than log tailing, at the cost of polling latency and load.
• Idempotent Consumer — since relays deliver at least once, the same message can arrive twice. Consumers must detect and absorb duplicates so that processing a message twice has the same effect as processing it once.

What this module covers

We start with the table that makes the write atomic (Transactional Outbox), then look at the two ways to drain it (Transaction Log Tailing and Polling Publisher), and finish on the receiving end, where at-least-once delivery forces consumers to be idempotent. Each lesson follows the same shape: the situation that leads to the pattern, the forces in tension, what the pattern does, a diagram, runnable example code, and an honest look at what the pattern costs you.

Related patterns

• Saga — depends on reliable event publishing for every step.
• Event Sourcing — another way to make the event the same write as the state change.

🧩 Two writes, one truth

The whole module rests on one move: never trust two independent writes to both happen. Pick one system — the database — as the source of truth, record the message there in the same transaction, and treat delivery as a separate job that can be retried until it succeeds. Everything else is detail.

What is the dual-write problem?

Two services writing to the same table at once A service must update its database and publish a message atomically, but they are separate systems with no shared transaction Writing the same row twice in one transaction A database that requires every write to be duplicated for backup

If a service commits to its database and then crashes before publishing, what happens?

The database change is rolled back automatically The message is published twice The change is durable but the rest of the system never hears about it — a lost event Nothing — the broker reconstructs the message from the database

What is the shared insight behind every pattern in this module?

Use two-phase commit between the database and the broker Make the message part of the same local transaction as the business change, and deliver it as a separate retryable step Publish the message before writing to the database Store all messages in the broker and never in the database

Submit

Mark as complete