A Backup Cannot Rewind the Business

My concern with financial data governance comes from incidents where recovery reached its limits.

At one company, an incorrect database write had widespread consequences and caused serious reputational damage. Restoring a recent backup offered a way to recover an earlier state, but it could not undo what had already happened to the business.

In another, a software release allowed customers to perform activities, including trades, that were not correctly recorded in a portfolio management module. Restoring data did not fill those gaps. Activity had continued across connected systems, and their interdependencies made timely reconstruction difficult. Some customer data could not be recovered.

Leadership had been confident that backups provided sufficient protection: if something went wrong, the system could go back in time and start again.

The database could return to an earlier state. The business could not.

Customers had acted. Other systems had processed transactions. Financial consequences existed outside the restored database. Recovering a trustworthy picture required evidence of those activities and a way to reconcile them.

These experiences changed what I expect from financial architecture. Preserving current state and backing it up are necessary capabilities. They do not, by themselves, establish that an organisation can explain its data or recover from incorrect processing.

For technology leaders, that distinction belongs at the centre of data governance.

The failures that leave systems running

Some of the most difficult integrity failures do not cause an outage.

I have seen problems remain unnoticed until a customer reported an incorrect portfolio. The application was available, yet the organisation could no longer confidently explain the result it displayed. Without sufficient history, even identifying the responsible subsystem became a difficult investigation.

In another incident, a service had a rounding defect. Small differences accumulated over time until their effects became significant. We lacked the historical detail needed to reliably recalculate the affected values and understand how the discrepancy had developed.

A backup could preserve those incorrect values just as faithfully as correct ones.

These incidents exposed different weaknesses: missing activity, delayed detection, and insufficient evidence for investigation. Treating them all as backup problems would have missed their causes.

Financial data governance must cover the full path from recording activity to detecting discrepancies, explaining results, and verifying corrections. Recovery is one part of that responsibility.

Preserve facts, govern their interpretation

In a financial SaaS platform whose development I supervised, auditability and reconstruction were principal reasons for choosing event sourcing.

We retained business events and built projections that interpreted them into application state. When processing logic changed, we replayed the relevant history to rebuild those projections. When a business action required correction, compensating events allowed us to preserve the original activity alongside the correction.

This created a useful separation between recorded facts and the logic used to interpret them.

The feature teams remained responsible for the correctness of their projection logic. They tested scenarios against expected results, and we manually checked some of the most important projections. Replay gave us a mechanism for applying logic to preserved evidence; it did not prove that the logic was correct.

That distinction matters to executives. A platform team can provide reliable storage, ordering, and replay mechanisms. Business and engineering owners must still agree on what the results mean and how their correctness will be established.

A calculation can be reproducible and consistently wrong.

An event store cannot recover an event that was never recorded

The incident involving missing trades also exposes a limit of event sourcing.

If an action occurs but no durable record captures it, replay has nothing to work with. Adopting an event store would not automatically resolve that failure.

The architecture must establish when an action is considered accepted, what evidence is durably recorded, and how activity performed in other systems is reconciled. An uncertain response from a connected system must have an explicit resolution path. Otherwise, the organisation may be unable to distinguish an action that failed from one that succeeded but was not recorded locally.

This is also why event-driven design and event sourcing should not be treated as interchangeable.

Event-driven communication distributes information between systems. Event sourcing establishes a retained history from which a system derives state. An integration feed may be useful for notifications without being complete enough to reconstruct a financial position.

Several consumers can process the same events and produce different results. They may use different rules, handle corrections differently, or have reached different processing positions. They can also agree while sharing the same missing input.

Leaders therefore need clear ownership of business facts and explicit guarantees about event meaning, completeness, and retention.

The evidence extends beyond the database

Some events in our platform referenced files in object storage. Those files were required to process the events, so they were part of the reconstruction history.

Every upload received a unique filename. Files were immutable. When a user replaced a file, we stored a new object and recorded a new event while retaining the original.

Preserving the event without preserving the referenced file would have left an incomplete record.

The same reasoning applies to other dependencies. A calculation may depend on a price, an exchange rate, a configuration value, or a particular business rule. If replay silently substitutes today’s value for the original input, it may produce a different result without explaining why.

The preservation boundary must include whatever evidence the organisation needs to support its reconstruction promises.

That does not mean retaining every technical detail indefinitely. It means deciding which facts and dependencies are essential, who owns their retention, and how their continued usability is verified.

Decide what replay is supposed to reproduce

“Can we replay it?” sounds like a single question. In practice, it can mean two different things.

One question is: what should the result be when we apply corrected logic to the historical inputs?

Another is: what did the system calculate and report at the time?

The first supports remediation. The second supports investigation and audit. They may require different records.

Reproducing an earlier result can require historical rules, configuration, and valuations, or a retained copy of the issued result. Recalculating with corrected logic requires a controlled explanation of why the new result differs.

The distinction becomes particularly important when a change introduces a new business policy. Applying that policy retrospectively is a business decision. It should not happen merely because a deployment changed the projection code and someone initiated a rebuild.

Auditability therefore requires more than a sequence of events. It requires a connection between evidence, interpretation, authority, and consequence. For a correction, that includes what was corrected, why, who authorised it, and which results were affected.

Make consistency a specific promise

Consistency is often discussed as though it were one property. Financial systems need several distinct assurances.

Replaying a history should agree with incremental processing under equivalent logic and inputs. Systems representing the same financial concept should reconcile at comparable cutoffs. Business rules should hold even when messages are delayed or retried. Records should also be checked against independent evidence where available.

Each assurance needs its own controls.

A system can satisfy replay consistency while omitting a transaction that occurred elsewhere. Two systems can disagree temporarily because one is behind. Two calculations can agree because they share a defect.

The useful governance question is therefore: which consistency promise applies to this result, and how do we know it has been met?

A critical financial figure should have an owner, a defined meaning, identifiable inputs, and an understood processing position. When those conditions are uncertain, the organisation needs a policy for what the system may continue to display or authorise.

Turn recovery confidence into evidence

The strongest assurance is a demonstrated capability.

For critical systems, I would expect teams to show that they can reconstruct state from preserved inputs, compare rebuilt results with incremental processing, and explain discrepancies. I would also expect them to demonstrate how they identify the customers and periods affected by a defect.

That exercise should include corrections, delayed activity, duplicates, and unavailable dependencies. Rebuilding must not accidentally repeat external actions. The resulting data must be validated before it becomes authoritative.

Successful reconstruction alone is insufficient. Results still need financial validation and reconciliation. Scenario tests, independent calculations, accounting controls, and targeted manual review provide different forms of evidence.

The leadership responsibility is to ask what each control establishes and where uncertainty remains. A green backup dashboard answers a narrow operational question. It does not demonstrate that a defective release can be investigated and its effects repaired.

An institutional capability

My experience has made me favour event sourcing where authoritative history, auditability, and reconstruction are central requirements. Other designs can provide these guarantees through ledgers, journals, versioned records, and disciplined recovery processes. The obligation is to demonstrate the guarantees.

This work has a cost. Preserved history requires storage and retention decisions. Replay requires maintained code and operational capacity. Auditability requires explicit ownership and correction procedures. These responsibilities should be funded and reviewed as part of the financial system itself.

The alternative cost often appears during an incident, when the organisation depends on individual engineers to infer what happened from incomplete evidence.

A trustworthy architecture turns that personal knowledge into an institutional capability. It allows the organisation to establish what it recorded, explain how it reached a result, identify what went wrong, and verify the repair.

The question I would put to any technology leadership team is this:

If a defect has been silently affecting customer data for a month, can we identify everyone affected, reconstruct the correct outcomes, and explain the corrections, or can we only restore a backup?