5 Times Backups Saved the Day

Backups are a vital part of a company’s IT strategy, as all business-critical operations involving technology can easily be impacted by a disaster. The ability to restore data and allow normal operations to resume in a timely manner can be the difference between bouncing back from a problem and closing your doors for good.

Prior to joining the team at Valiant in 2015, I spent 15 years designing and maintaining networks, and leading IT departments for various companies. From non-profit organizations to fast-paced publishing companies, the proper implementation and maintenance of backup strategies kept business running, even in the wake of a disaster.

Here’s a look back at some of the problems I’ve faced, and how proper backups saved the day.

Lessons Learned From 9/11

I was only 20 years old when 9/11 happened and had been working in IT for less than a year. My office was located in Manhattan’s financial district, just a few blocks away from ground zero, and we weren’t able to enter our building for a week.

The organization’s CIO was injured, and that left me with the responsibility of getting us back up and running, and communicating with the public, as soon as possible. Tape-based backups were the go-to-choice for backups at the time, and we kept copies in 2 external locations: a safe deposit box at a local bank, and a fire-proof safe in my home.

While I was not able to access the office, or the bank where we stored our tapes, I was able to restore data from the tapes at my home on to a series of CDs and copy the data to servers that we had in a datacenter that was reachable. Within a few hours, I was able to build a server that let the staff remotely access data that enabled communications with the individuals we served, our vendors, and other contacts.

This really wasn’t the greatest approach to take, but it did help resume operations quicker than if I did not have the tapes – and it paved the way for a greater understanding of how to handle disaster scenarios for a business.

The Blackout of 2003

Fast forward to the blackout of 2003, and we were prepared to handle a disaster that halted business operations for many other organizations for days. The revised backup strategy, along with a disaster recovery plan, put in to place in response to the lessons learned during 9/11 kept the business running in the middle of a blackout that affected the entire northeast.

The redundant network on the west coast was activated within minutes of the blackout, allowing services like email and our business-critical applications to continue running and were accessed by staff in other parts of the country with little to no interruptions, allowing the NY headquarters to recover from the event while business continued as normal outside of the area.

Demagnetized Backup Tapes

As an organization grows, so do its technical requirements. Additional resources such as increased storage is needed to house more data are needed, and more data often means more time is needed to properly perform and verify backups.

Just as the organizations technical requirements changed, so did the technology used to support them. We switched between several tape backup formats over the years, transitioning from DLT to LTO and then to VXA – also opting to have our tapes stored in a secure facility at the same time.

The new backup system worked like a charm – until it didn’t. Our first need to restore data from the VXA tapes was a nightmare.

There’s absolutely nothing on these tapes. It’s like they’ve never been used.

I remember looking at the organization’s CIO and saying just that. It was like the tapes that we backed data up to and verified were replaced with tapes that were brand new.

This problem made light of an earlier concern around the new backup system – backup verification seemed to be taking much less time than expected, and a software incompatibility was the identified cause.

Despite prior testing of the system, I wasn’t 100% convinced that it was stable and insisted on continuing backups with our previous system for several months as a backup, and our backup of the backup saved the day.

Phone System up in Flames

We tend to concentrate on protecting data that is mission-critical; accounting records, project-related files, and other information that is required for the day-to-day operations of a business. While protecting this data is important, it’s equally important to consider information stored in unlikely places that may otherwise go unprotected.

A different company I worked for in the past had aging phone system that experienced a massive failure overnight, with its T1 card catching fire and destroying all of the related hardware. While the telecommunications vendor that managed the system was incredibly responsive and had a replacement system up and running within 24 hours, there was one detail missing that prevented operations from being restored as soon as possible – they had no backup of the phone system’s configuration.

Without a proper configuration to operate with, we had a brand-new phone system that didn’t function. Thankfully, a monthly backup of the phone system’s configuration was a part of my backup strategy and the phones were ringing as usual within minutes of applying it to the new system.

Redundancy is Not a Form of Backup

Several years later we had a similar failure that involved a fire – this time it was the company’s accounting server.

The server was an absolute workhorse and was configured to handle large amounts of usage with ease and a high level of fault tolerance; redundant power supplies on a dedicated power circuit, a RAID5 disk array with a hot spare to reduce downtime in the event that a disk experienced a failure, and even a dedicated backup power unit with environmental sensors.

One of the server’s power supplies failed and caught fire overnight. The fire damaged the server beyond repair, and none of the built-in redundancies were of any help due to the extent of the damage. The power supply that caught fire damaged the backup power supply and the entire RAID system was destroyed. The fire suppression system in the server room prevented the spread of damage, but the accounting server was nothing more than metal and melted plastic by the time the fire was out.

While the server was a complete loss, the data stored on it was backed up to another server and transferred to an off-site location nightly. A new server would arrive 2 days later, but 2 days without accounting activity can have a large impact on a company, so I restored the accounting applications and data to another server, temporarily, keeping the accounting staff working with minimal interruption until their new server was in place.

All of the disaster scenarios above support businesses of all sizes having data backup plans in place. You may not be able to schedule a disaster, but you can prepare your business to continue in the wake of one. Are you running your business without backups? Get in touch today and our team will work with you to understand your needs and craft a backup strategy that will keep your business running smoothly.