Distributed Data Consistency: The Saga Pattern and Eventual Consistency

Where did the 'transaction' go in a distributed system?

It could not cross the boundary: a single database's ACID transaction does not span service or shard boundaries — and protocols that try, with distributed locks, take availability hostage under high traffic. The modern answer splits the long transaction into local steps and manages failure through compensation: the saga pattern. This article builds saga's two forms, compensation design, and how eventual consistency is explained to the user.

Saga: local transactions + a compensation chain

A saga splits a multi-service business flow (place order → reserve stock → charge payment → plan shipping) into steps, each a local transaction in its own service. When a step fails, undo happens not by database rollback but through compensations: if the charge fails, the reservation is released and the order canceled. It comes in two fundamental forms, and the choice shapes the architecture's character.

Orchestration or choreography?

The practical rule: orchestration for backbone flows like order/payment (with a queryable saga record), choreography for side effects like notifications and loyalty. To keep the orchestrator from becoming a 'distributed monolith brain', it carries no business rules — it only manages the flow; rules stay in the services.

Compensation design: put the irreversible last

Compensation is not mathematical inversion — 'email sent' cannot be unsent, a charge is only compensated by refund. Step ordering is therefore a design decision: easily reversible steps first, irreversible steps (money capture, notifying external systems) as late as possible. Additional disciplines: every step and its compensation must be idempotent (no double effects when the saga retries — our async article's rule applies here too); compensations themselves can fail, so a compensation queue + alert + manual runbook is required; and every saga has a timeout — a flow stuck 'half-done' forever is a ghost that locks inventory.

Eventual consistency: designing the window for users

Because steps flow asynchronously, the whole is not instantaneously consistent — and its user-facing surface belongs to experience design, not just engineering. Patterns that work: state transparency ('your order is received, awaiting payment confirmation' — visible steps, no ambiguity); optimistic confirmation + graceful walk-back (the order is accepted instantly; in the rare failure, an apologetic notification + automatic refund — saving seconds at campaign scale); read-your-writes protection (the user's own action appears instantly on their own screen — the read-from-primary rule from our database article); and avoiding false negatives (designs that wait out the propagation window before declaring 'not found'). The window's duration is measured and shared with the product team — 'how late can it be?' deserves a metric, not a guess.

Reconciliation: eventual consistency's insurance

Events get lost, compensations fail, integrations produce duplicates — that is distributed reality. The insurance is the discipline familiar from our ERP integration article: periodic reconciliation — cross-system record counts and sampled comparisons, automatic difference reports and a correction procedure. Saga records are gold here: the list of 'started but unfinished' flows is reconciliation's first query. For financial flows the report is daily; weekly is the practical rhythm elsewhere.

Business impact: the consistency model is a product decision

Insisting on strong consistency in a distributed system sacrifices either scale or availability; uncontrolled looseness destroys trust with double charges and lost orders. The right point is chosen per business flow: money movement 'definite and traceable', notifications 'eventually' is fine. Documenting that choice — which flow runs under which guarantee — gives clarity to both engineering and customer communication: on campaign day, 'did my payment go through?' has an answer, because saga state is queryable.

Frequently asked questions

Why not two-phase commit (2PC)?

Resources locked on coordinator failure and dependence on all participants' simultaneous health are incompatible with high traffic. The practical cross-service standard is saga + compensation; 2PC remains for exceptional, controlled internal scenarios.

Where should saga state live?

In orchestration, in the saga manager's own table (step, status, attempts, last error); in choreography, in each service's processed-event record plus an end-to-end correlation ID. In both, 'where is this flow?' must be answerable with one query.

How do outbox and saga relate?

Outbox guarantees one service's 'write + publish' atomicity; saga manages many services' flow integrity. Saga steps publish their events through outboxes — layers, not rivals.

How is this tested?

Contract tests for steps and compensations + fault-injected integration tests of the flow ('what if step 3 fails?') + chaos drills. The compensation path must not run for the first time in production.

Distributed consistency checklist

1. Consistency requirement (strong/eventual) written per flow

2. Backbone flows use orchestrated sagas; state queryable

3. Step order: irreversibles last; all steps+compensations idempotent

4. Compensation-failure queue, alerts and runbook in place

5. Consistency window measured; UX designed with state transparency

6. Read-your-writes protected on critical screens

7. Periodic reconciliation reports and half-flow queries running

SSH Yazılım designs saga-based distributed workflows and consistency architecture end to end. Let us secure your flow integrity together.