Compensation Is Not Rollback

Distributed systems don't get an undo button.

Something fails halfway through a workflow, and somebody in the room says "we'll just roll it back."

Everyone nods, because rollback is one of the few things in software that's genuinely simple. You've used it a thousand times. It works.

Then you remember the order went through five services and three external APIs. The card was charged eleven seconds ago, by a payment provider that has never heard of your transaction.

What rollback actually is 🔗

Rollback is a database primitive. That's not a technicality, it's the whole point.

When a transaction fails, the database reverses every change inside it, atomically. There's no partial state to inspect and no half-committed row to clean up. The system returns to exactly where it was, and no observer ever sees the intermediate steps.

This works because one system controls all of the state. The database holds the data, the log, the locks, and the clock. It can promise atomicity because it never has to ask anyone else for permission. Rollback is instant, safe, and complete. It's complete only inside that boundary.

Everything people believe about rollback is true. It's just true about one database.

Why distributed systems break the promise 🔗

Now spread the same workflow across services.

Service A charged the card. Service B reserved the inventory. Service C sent the confirmation email. Service D fails.

Each of those services committed its own change, independently, to its own store. Each was right to. There was no shared transaction, because there's no practical way to have one. Two-phase commit exists, it doesn't scale, and it turns every participant into a hostage of the slowest one.

So there's nothing to reverse. You can't un-send an email. You can't un-charge a card atomically, because the charge lives in a payment provider's ledger and that ledger isn't yours. The inventory reservation is already visible to other customers.

The only direction available is forward. Not back to the previous state, but on to a new one that resembles it.

What compensation actually is 🔗

A compensating action is a new business action that semantically reverses a previous one. That word semantically carries real weight.

You issue a refund. That isn't an un-charge. It's a second transaction, leaving two entries on the customer's statement and two rows in the provider's ledger. You cancel the reservation, which is a new state rather than the absence of the old one. You send a cancellation email, which sits in the inbox underneath the confirmation you wish you hadn't sent.

The end state is close to where you started. It isn't identical, and the difference isn't cosmetic. Anyone reading your data later will see that something happened and then something else undid it. An auditor, a support agent, a customer. That history is real, and it's often exactly what the business wants.

Compensation is an operation like any other in your system. It runs business logic. It calls services. It writes to databases. And because it does all that, it can fail.

The four differences that matter 🔗

Rollback is atomic. Compensation is eventual. There's a window, sometimes a long one, where the charge has happened and the refund hasn't. Your system is visibly inconsistent during that window, and you have to decide what it shows users.

Rollback is invisible to the business. Compensation is a business event. Nobody outside the database learns about a rolled-back transaction. Everybody learns about a refund, including your finance team.

Rollback cannot fail. Compensation absolutely can. The refund API can be down. The cancellation can time out. The inventory service can reject the release, because someone else already bought the item.

Rollback is symmetric. Compensation is often asymmetric. Charging takes one call. Refunding takes a call, a settlement delay, and sometimes a manual review. The undo is rarely a mirror image of the do.

Treating compensation as if it had rollback's properties is the most common distributed systems mistake I see. It stays invisible until the day it gets expensive.

Design compensation on purpose 🔗

The practical work is unglamorous, and it pays for itself.

Go through your workflow step by step. For each one, write down what undoing it means as a new action. Not "reverse the payment step" but "call the refund endpoint with the original payment reference and record the refund id". If you can't write that sentence, you don't have a compensating action. You have a hope.

Model those actions as first-class workflow steps. They deserve the same retries, logging, idempotency keys, and monitoring as the forward path. Compensation tucked into a catch block gets none of that, and it's the code that runs on your worst day.

Then mark the steps you can't compensate at all. Sending an SMS is the honest example. It's gone, it's on someone's phone, and no API retrieves it. Once you know which steps are one-way doors, you can move them as late in the workflow as possible. That costs nothing and saves a great deal.

Finally, the question almost nobody asks. What happens when the compensation itself fails? You need an answer, and it's usually retries, an alert, and a human. A failed compensation is its own error case, and it's the one that leaves money in the wrong place.

The saga pattern is this, written down 🔗

Sagas are compensation made explicit. A saga is a sequence of local transactions. Each step has a defined compensating transaction, and if step N fails, the compensations for the preceding steps run in reverse.

That's the whole idea. The value isn't the mechanism. It's the discipline of naming the undo for every step at design time, instead of at three in the morning.

You get two flavours. Choreography, where services react to each other's events and the workflow exists only as an emergent property of who listens to what. Orchestration, where a coordinator holds the state and tells each participant what to do next. Choreography couples less and is harder to see. Orchestration is easier to reason about and puts the logic in one place.

Both need the same thing underneath: reliable messaging. A saga running on delivery you don't trust is a state machine that silently skips states, which is worse than no saga at all.

What this means in practice 🔗

• Stop saying rollback about anything that crosses a service boundary. The word imports guarantees you don't have.
• Write the compensating action for every step, as a concrete call with concrete arguments.
• Give compensations the same engineering as the forward path, including idempotency, because they'll be retried.
• Identify the steps you can't undo and push them as late in the workflow as the business allows.
• Plan for a failed compensation. Retries, alerting, and a human path for the ones that stay broken.

Closing 🔗

The mental model that causes the damage is the one where distributed workflows are just longer database transactions. They aren't. They're a series of small commitments to the outside world, each becoming real the moment it happens.

Compensation is a business action. Rollback is a database primitive. Model them differently, because they're different, and the gap between them is where the money goes missing.

Which step in your workflow has no real compensating action? That's usually the interesting one. Come tell me about it on Twitter/X.

saga-pattern compensation distributed-transactions messaging distributed-systems