I keep all sensitive data on versioned cloud storage. Having been through HDD death several times (even on RAID arrays) I’d rather have the threat of third-party compromise rather than the threat of fire, flood, or malicious actors. However, unless you practise the restore process, off-site backups can be practically useless…:
Browse the comments on virtually any story about a ransomware attack and you will almost surely encounter the view that the victim organization could have avoided paying their extortionists if only they’d had proper data backups. But the ugly truth is there are many non-obvious reasons why victims end up paying even when they have done nearly everything right from a data backup perspective.
This story isn’t about what organizations do in response to cybercriminals holding their data for hostage, which has become something of a best practice among most of the top ransomware crime groups today. Rather, it’s about why victims still pay for a key needed to decrypt their systems even when they have the means to restore everything from backups on their own.
Experts say the biggest reason ransomware targets and/or their insurance providers still pay when they already have reliable backups is that nobody at the victim organization bothered to test in advance how long this data restoration process might take.
“In a lot of cases, companies do have backups, but they never actually tried to restore their network from backups before, so they have no idea how long it’s going to take,” said Fabian Wosar, chief technology officer at Emsisoft. “Suddenly the victim notices they have a couple of petabytes of data to restore over the Internet, and they realize that even with their fast connections it’s going to take three months to download all these backup files. A lot of IT teams never actually make even a back-of-the-napkin calculation of how long it would take them to restore from a data rate perspective.”
Wosar said the next most-common scenario involves victims that have off-site, encrypted backups of their data but discover that the digital key needed to decrypt their backups was stored on the same local file-sharing network that got encrypted by the ransomware.
The third most-common impediment to victim organizations being able to rely on their backups is that the ransomware purveyors manage to corrupt the backups as well.
“That is still somewhat rare,” Wosar said. “It does happen but it’s more the exception than the rule. Unfortunately, it is still quite common to end up having backups in some form and one of these three reasons prevents them from being useful.”
Bill Siegel, CEO and co-founder of Coveware, a company that negotiates ransomware payments for victims, said most companies that pay either don’t have properly configured backups, or they haven’t tested their resiliency or the ability to recover their backups against the ransomware scenario.
“It can be [that they] have 50 petabytes of backups … but it’s in a … facility 30 miles away.… And then they start [restoring over a copper wire from those remote backups] and it’s going really slow … and someone pulls out a calculator and realizes it’s going to take 69 years [to restore what they need],” Siegel told Kim Zetter, a veteran Wired reporter who recently launched a cybersecurity newsletter on Substack.
“Or there’s lots of software applications that you actually use to do a restore, and some of these applications are in your network [that got] encrypted,” Siegel continued. “So you’re like, ‘Oh great. We have backups, the data is there, but the application to actually do the restoration is encrypted.’ So there’s all these little things that can trip you up, that prevent you from doing a restore when you don’t practice.”
Wosar said all organizations need to both test their backups and develop a plan for prioritizing the restoration of critical systems needed to rebuild their network.
“In a lot of cases, companies don’t even know their various network dependencies, and so they don’t know in which order they should restore systems,” he said. “They don’t know in advance, ‘Hey if we get hit and everything goes down, these are the services and systems that are priorities for a basic network that we can build off of.’”
Wosar said it’s essential that organizations drill their breach response plans in periodic tabletop exercises, and that it is in these exercises that companies can start to refine their plans. For example, he said, if the organization has physical access to their remote backup data center, it might make more sense to develop processes for physically shipping the backups to the restoration location.
“Many victims see themselves confronted with having to rebuild their network in a way they didn’t anticipate. And that’s usually not the best time to have to come up with these sorts of plans. That’s why tabletop exercises are incredibly important. We recommend creating an entire playbook so you know what you need to do to recover from a ransomware attack.”