When a service updates its database and wants to notify the rest of the system by emitting an event to Kafka, RabbitMQ or SQS, the naive code looks like this: write to the database, then publish. If publishing fails after the commit, the event is lost. If publishing succeeds but the commit fails, the event refers to a state that does not exist. Both cases define the dual-write problem.
The Transactional Outbox pattern fixes that inconsistency with a simple idea: never publish directly. The event is written to an outbox table within the same SQL transaction as the business change. A separate process reads that table and publishes to the broker. As long as the SQL transaction is atomic, the database and the future event are consistent by construction.
This second article in the distributed architecture patterns series follows the Saga pattern, which it naturally completes for steps that must emit events reliably.
The dual-write problem: two writes without atomicity
Back to the e-commerce order example. When the status moves to confirmed, several systems must be notified: the email service, the analytics service, the data warehouse. We publish a OrderConfirmed event to Kafka.
def confirm_order(order_id: int) -> None:
with transaction.atomic():
Order.objects.filter(id=order_id).update(status="confirmed")
kafka.publish("orders.confirmed", {"id": order_id})
Four cases exist:
- The transaction commits, publishing succeeds. The nominal case.
- The transaction fails, publishing does not happen. Consistent.
- The transaction commits, publishing fails (broker down, timeout). The order is confirmed in the database but nobody knows. The email never goes out.
- The transaction commits, publishing succeeds, but the network drops before the ack. The worker retries and publishes a second time. Duplicate.
You can swap the order (publish first, commit later), but that just turns case 3 into the opposite case: an event that refers to an order that was never confirmed. The problem does not vanish, it changes shape.
No try/except combination fixes this. You need a mechanism that makes publishing itself transactional.
The principle: one transaction for two destinations
The Outbox idea: publishing does not write to Kafka, it writes to an SQL outbox table. That write is part of the business transaction. If the commit succeeds, the event exists in the database. If it fails, the event never existed.
A separate process, called the relay or dispatcher, reads the outbox table, publishes each row to the broker, then marks the row as published (or deletes it). That process can crash, restart, retry as much as it wants. As long as it publishes at least once whatever it finds in the table, the event arrives.
The system becomes at-least-once: a consumer may receive the same event several times. That guarantee is acceptable if consumers are idempotent. In practice, it is the only guarantee reachable in a distributed system, and far more useful than the “exactly-once” no one ever truly has.
The outbox table
A minimal schema:
from django.db import models
from django.db.models import Q
class OutboxEvent(models.Model):
id = models.BigAutoField(primary_key=True)
aggregate_type = models.CharField(max_length=64)
aggregate_id = models.CharField(max_length=64)
event_type = models.CharField(max_length=128)
payload = models.JSONField()
created_at = models.DateTimeField(auto_now_add=True)
published_at = models.DateTimeField(null=True)
class Meta:
indexes = [
models.Index(
fields=["id"],
name="outbox_unpublished_idx",
condition=Q(published_at__isnull=True),
),
]
The partial index only contains unpublished rows. On a table that keeps growing (millions of archived rows), it stays close to zero size and the relay query stays constant. Django has supported this syntax through condition= since version 2.2 and translates it into CREATE INDEX ... WHERE on PostgreSQL. The monotonic id guarantees a stable publication order per aggregate.
The write becomes:
def confirm_order(order_id: int) -> None:
with transaction.atomic():
Order.objects.filter(id=order_id).update(status="confirmed")
OutboxEvent.objects.create(
aggregate_type="order",
aggregate_id=str(order_id),
event_type="OrderConfirmed",
payload={"id": order_id},
)
No more kafka.publish in business code. The transaction is purely SQL, atomic end to end.
The relay: publishing what was written
The relay loops, reads the unpublished events, sends them to the broker, and marks them.
from celery import shared_task
from django.db import transaction
from django.utils import timezone
BATCH_SIZE = 100
@shared_task
def relay_outbox() -> None:
with transaction.atomic():
events = list(
OutboxEvent.objects
.select_for_update(skip_locked=True)
.filter(published_at__isnull=True)
.order_by("id")[:BATCH_SIZE]
)
if not events:
return
for event in events:
kafka.publish(
topic=f"{event.aggregate_type}.{event.event_type}",
key=event.aggregate_id,
value=event.payload,
)
OutboxEvent.objects.filter(
id__in=[e.id for e in events]
).update(published_at=timezone.now())
Three important design choices hide in there.
select_for_update(skip_locked=True) lets multiple relay workers run in parallel. Each worker grabs a batch that the others skip. That is what makes horizontal scaling possible without internal concurrency duplicates.
Publishing happens inside the transaction that marks published_at. If Kafka acks but the transaction then fails, we republish on the next round. At-least-once by design. skip_locked keeps that slightly long lock from blocking other workers: they simply skip the locked rows and pick a different batch.
Ordering by id guarantees that events for the same aggregate are published in the order they were created, provided the topic is partitioned by aggregate_id (Kafka key).
The task processes a single batch and returns. Celery Beat calls it again every few seconds. That keeps transactions short, avoids task timeouts, and leaves the worker free between passes.
Consumer-side idempotency
The contract is at-least-once. Consumers must be idempotent. Two approaches.
Deduplication by event_id. Add a stable UUID to OutboxEvent and each consumer keeps a processed_events(event_id) table with a unique index. Before processing, it tries the insert. If it fails (unique constraint violation), the event has already been processed.
Naturally idempotent operation. If the consequence of the message is “set status to confirmed”, replaying has no effect. Simplest case, and the one to aim for whenever the design allows it.
Without consumer-side idempotency, the Outbox pattern produces silent duplicates. It is a discipline to enforce from the first consumer.
Outbox + Saga: the natural combination
A Saga step that needs to publish an event uses the Outbox without doing anything else. The step’s business write and the event to emit share the same transaction.atomic. The compensation, if it runs, also writes its own event to the Outbox (PaymentRefunded, OrderCancelled), strictly symmetrically.
Without the Outbox, a Saga step that publishes directly to Kafka may succeed on the publish and fail on the DB commit. The ecosystem believes an order was created when it was not. The two patterns reinforce each other: the Saga handles workflow consistency, the Outbox guarantees that step events are emitted if and only if the step actually happened.
CDC: the alternative without an application relay
A variant of the pattern delegates the relay to an external tool based on Change Data Capture. Debezium reads the PostgreSQL WAL, sees inserts into the outbox table, and publishes to Kafka. No application code for the relay.
Pros: no application load, lower latency (the WAL is read in near real time), no polling. Cons: an extra component to operate (Kafka Connect, Debezium), more complex configuration, and a coupling to PostgreSQL or MySQL through their replication mechanisms.
For a modest Django project, a Celery relay is more than enough. For a high-throughput system with tight latency requirements, CDC becomes relevant.
Common pitfalls
Not purging the Outbox. The table grows forever if nothing is deleted. Either you delete rows after publishing, or keep them a few days and then archive. Keeping history has audit value, but an outbox containing millions of rows eventually slows down the unpublished selection despite the index.
Confusing payload and current state. The event payload must reflect the state at write time, not the current state read at publication time. If the relay reads Order on the fly, it publishes the present state, which may already have changed. Serializing the payload in the original transaction avoids that pitfall.
Forgetting aggregate-based partitioning. Without key=aggregate_id on the Kafka side, two events for the same aggregate may land on different partitions and arrive out of order on the consumer side. Kafka’s ordering guarantee is per partition, not global.
When not to use the Outbox
The pattern adds a table, an index, a worker, and a consumer-side idempotency discipline. For a monolithic application that publishes nothing externally, it is useless.
The Outbox becomes relevant once:
- you publish to an external broker (Kafka, RabbitMQ, SQS, Redis Streams)
- consistency between your database and the published event is critical
- you want to survive broker outages without losing events
- you combine it with a Saga or another long-running workflow
Conversely, for a notification where loss is acceptable (an analytics log, a monitoring ping), publishing directly is simpler and largely enough.
Conclusion
The Transactional Outbox turns a distributed problem (two writes to coordinate) into a local one (a single SQL transaction). All the hard work is delegated to a relay that can crash, restart and retry without risking consistency.
Like the Saga, it is a pattern whose usefulness reveals itself the moment you start observing what actually happens in production. The day a broker is down for thirty seconds and no event is lost, the cost of an extra table feels trivial.