Home/Blog/Custom software

The backup no one had tested

They had a backup. The little light was green. The day they needed it, they found it had spent seven months saving an empty folder.

Inma's accountancy practice, in a large town in Toledo, central Spain, had a backup. Inma said so calmly when anyone asked. "We do backups." There was a small device in a corner of the office, with a little green light, that switched on every night and made a small noise. It had been there for years. The light was green. Everyone slept soundly.

The day the main computer wouldn't turn on was a Tuesday in February. No signs of life. The local IT guy, the one who fixed their things, looked at it and said the disk had died, just like that, no warning, the way disks die. "It's fine," said Inma, "we have a backup." And they went to the corner, took the device with the little green light, and opened it to get out what was inside.

Inside there was a folder. And the folder was nearly empty.

What the little green light didn't say

What had happened was one of those things that happen and nobody sees coming. Seven months earlier, someone had moved the practice's working folder — all the client files, the tax returns, the payrolls — while reorganising the computer. The backup program kept doing its backup every night, on time, with its little green light. But it was copying the old folder, the one that had been left empty. For seven months, the device faithfully saved, every night, a folder where there was nothing left.

The little green light wasn't lying. It told the truth: "I've done the backup." What it didn't say, because it didn't know how to, was "I've backed up the wrong thing." Nobody ever checked, because checking a backup is one of those things you don't do while you don't need to, and by the time you need to, it's too late.

The practice lost seven months of work. Seven months of files, of calculations, of documents. Not everything, because part of it was in emails and on paper, but enough for three horrible weeks, calls to clients asking them to resend things, and a month's billing thrown into rebuilding what was already done.

A backup isn't a backup until it's restored

Inma called me afterwards, once the worst had passed, because she never wanted to live through that again. And the first thing we talked about was this, which is the only thing that really matters about backups and that almost nobody takes into account.

A backup is worth nothing until you've restored it. Until that moment it's a promise, not insurance. You can have a device, a little green light and a small noise every night for years, and have nothing, as Inma had. The only way to know whether a backup works is to take it, restore it somewhere else, and see whether what comes out is what was supposed to come out. And you have to do that now and then, not once when you install it and never again.

—So the backup I had wasn't a backup?
—It was a backup. What it wasn't was a tested backup. And an untested backup is like an empty fire extinguisher: it looks like it's there, until the day of the fire.

What we set up, which wasn't much

What we did at Inma's practice wasn't anything spectacular, and that's exactly the point. Reliability isn't spectacular. It's boring, and that's why it gets neglected.

We put backups in more than one place, because a single backup, on a single device, in the same office, goes with the office if there's a fire, a flood or a break-in. One backup on site and another off site, somewhere safe, automatic, without anyone having to remember to do anything.

And, above all, we added two things the little green light didn't provide. The first, an automatic check that every so often took the backup, tried to restore part of it, and flagged it if what came out wasn't what it should be. A backup that watches itself. The second, a real alert when something failed: not a light you have to go and look at, but a message that arrives on its own — "last night's backup doesn't include the clients folder" — on the day it happens, not seven months later.

And once a quarter, a real fire drill: restore everything on a separate machine and check that the practice could go back to working with it. One morning's work every three months. The price of really sleeping soundly, not of believing you sleep soundly.

What you actually buy

Inma asked me how much that cost a month, and when I told her, she laughed, but with relief.

—That's all? I spent more than that rebuilding a month of work.

I told her it's almost always like that. That reliability costs little compared with what it costs not to have it, but it costs beforehand, and that's why it gets put off. It's like car insurance: it looks like a useless expense every month, until the month of the crash.

And I told her one last thing, which is the one that really matters. What you buy with a properly made backup isn't a backup. It's being able to say "it's fine, we have a backup" and have it be true. Inma had said it for years believing it was true. Now she says it knowing it is. The difference between the two sentences, which sound the same, is seven months of work and three weeks I wouldn't wish on anyone.

Because with backups, as with so many things in a company, the problem isn't not having them. It's believing you have them. And that's only cured by checking.

Three questions for your backup

Has anyone ever tried to restore it? If the answer is no, you don't have a backup: you have a promise. A backup is only confirmed by restoring it and seeing that what comes out is what should.

Is there a copy off the premises? A single backup in the same place goes with the place in a fire, a flood or a break-in. Does it alert you when it fails, on the day it fails? A green light you have to go and look at isn't an alert. Reliability is boring, and that's why it gets neglected; it costs far less than the day you miss it.

When was the last time anyone tested your backup?

If the answer is "never" or "I don't know", it's worth checking before you need it. We can look together at how your company's information is protected.

Let's talk