Logo
Logo

Back Alive and Running

Vriti Magee | Nov 10th 2026

IMG_2026-11-11-101742.jpeg

"Once well and truly dead. Now back alive and running, and the till is ringing." Illustrated by DALL·E

It is 3 a.m. on a Saturday. The people are paged, the coffee is bad, and the first piece of news is good: the data is back. The immutable copies held. The vault did its job.

The application, however, is still dead.

Now the real work begins. Everyone who built the app is needed. Two are on holiday, which is going rather better for them. Someone runs the infrastructure-as-code and gets part of the estate back. Someone else opens the runbook. It was last updated six months ago, and a fifth of the environment has moved since.

Then come the questions, none of which have written answers. What comes up first? Is the cluster actually serving, or are the pods just running? And what about the security group added during last month's incident, the one that was never codified and never made it into the backup? The cluster is up. It is not talking to the database. Nobody is quite sure why.

"It has to be rebuilt live and that takes days and not hours."

Out of Stealth, Into the Scramble

Every new backup company must first face the obvious heckle: does the world really need another one? Dreeze, newly launched, sidesteps it. Backing up data, it concedes, is a solved problem, or near enough. Getting the application back is not. Two of the founders built recovery tools for the physical and virtual eras; the third was a CIO on the receiving end. Their bet is that the cloud broke recovery, even as backup matured.

The reason is anatomy. An application on premises was ten or twelve virtual machines, each neatly wrapping its own complexity. A cloud-native application is hundreds of resources: subnets, roles, keys, secrets, queues, functions, gateways, and the third-party services it talks to. The dependencies between them are many, sometimes circular, and they shift with every deployment.

"What was true last month isn't true now."

The knowledge of how it all fits together is scattered across code, runbooks and the memories of whoever happens to be awake.

The Question From the Board

Every CIO has been asked it, usually in a room with good chairs and a sombre mood. If we take a hit, how fast are we back on our feet? Is this going to be a bad day, or an incident that costs millions?

The honest answer most can give is that the data is safe. It is a true answer to a different question. Boards do not lose sleep over snapshots or storage volumes. They want to know whether the apps are running, taking orders and earning their keep. Restoring data takes hours. Reviving the application takes days, and it is improvised every time, because the environment keeps evolving.

"If you cannot look them in the eye and say with confidence yeah we know we can recover we've proven that, that is a very difficult thing."

Note the word proven. Not planned, not documented, not promised in a slide. Proven.

The Test That Never Happens

So the follow-up question is the awkward one: when did you last test it? One architect at a Fortune 500 airline put it plainly: the recovery test happens once a year, and it is "a very difficult exercise". A director of infrastructure at a global biotech went further: every recovery test is a project.

The logic is circular, in the worst sense. Recovery is manual, so testing it needs the same people, the same stale runbook and the same guesswork. So it is expensive. So it is rare. Teams spend a week preparing for the drill, which is a week more notice than any attacker has ever given. And a test passed six months ago describes an environment that has since moved on without telling anyone.

"If you're not testing enough you don't have an SLA, you have hope."

Hope is an admirable quality in a person. In a recovery objective, it is a liability with a nice font.

Presto and Green

The cure has two ingredients: frequency and depth.

Frequency first. Once a year is not enough in the cloud. Some applications deserve a quarterly test, some monthly, some weekly, and a few daily. That is only affordable if recovery is orchestrated rather than improvised, so a drill stops being a project and becomes a routine.

Depth matters more, because testing is not binary. A screen gone presto and green proves that things started, not that they work. The database came back and answers queries. Splendid.

"Now if I'm in a real scenario and I can do SQL queries how does that help my customers?"

A deep test asks what the business would ask. Can a customer still buy a product? Can a support ticket still be closed? The test runs in an isolated bubble, so nobody accidentally closes a real customer's ticket. It can also be trimmed: leave the live DNS alone, or restore 100 GB instead of a petabyte and run the drill a thousand times faster and cheaper.

🛠️ Architectural View: Readiness Gets a Number

Untested means unprotected.

An application is not considered protected until it has actually been tested.

Recovery risk, per application.

If any member resource is unprotected, the score is exactly 100: Protection is binary. If the latest test failed, it sits between 80 and 99. Otherwise it blends how stale the last test is with gaps against recovery point and time objectives, so one bad dimension cannot be averaged away.

No fake comfort.

A missing input returns null with a reason, never zero.

Impact, per resource.

Data sensitivity, environment and role combine into an importance score. Highest risk and highest impact get tested first.

Surprises surface early.

A circular dependency that would block recovery shows up in a drill on a Tuesday afternoon, not at 3 a.m. on a Saturday.

Closing Reflections

An untested recovery plan is a hypothesis. Boards are entitled to evidence. The uncomfortable truth is that most organisations can restore their data and cannot say, with a straight face, when they last brought an application back to life, end to end, and watched it generate revenue again.

There is a pragmatic footnote too. On the night, perfection is not on the menu: you recover whatever you can recover, rather than nothing because one piece is missing. Knowing in advance which piece is likely to go missing is precisely what frequent, deep testing buys.

A few questions worth taking into Monday morning:

  • When did you last revive, not restore, a critical application?
  • Could you close a support ticket in it afterwards?
  • Would your runbook survive 3 a.m., a holiday roster and last month's hotfix?

If the answers involve the word "hopefully", you have your answer.

In a follow-up piece, The Empty Queue, I will share my own take from a personal demo of the product, and why infrastructure-as-code, for all its virtues, is not a recovery plan.

🔍 Links for Further Reference

Watch the full Cloud Field Day 26 sessions:

Recent Articles