Disaster recovery planning sits as high on the management agenda as any kind of major operational or business development decision. But just having a continuity plan in place is not enough. As political and environmental environments shift rapidly, risks to the supply chain should be regularly looked at and your disaster recovery plan tested (and adapted if necessary) to ensure it more than carries muster should disaster strike.  

John Nicholson, our Head of Infrastructure and Service Delivery looks at potential risks to your supply chain IT system that all businesses should be aware of, and discusses what makes a good disaster recovery plan.

Why continuity planning for the supply chain is so important 

For businesses dependent on their IT systems to operate fully, having a disaster recovery plan in place is crucial. It’s incredibly important for businesses such as ours as a supply chain management technology provider to many global businesses who are reliant on our systems. We operate all of our IT equipment such as servers, networking and data storage from a third-party datacentre. If the datacentre experienced an unrecoverable disaster such as a fire, then that could potentially mean all of our data (and our customers’ data) is destroyed. This is an unacceptable scenario for us and with an active and well-maintained disaster recovery plan we substantially mitigate this risk by ensuring multiple copies of the data are stored safely off-site at regular intervals throughout the day. Disaster does not have to mean disruption. 

Time is of the essence in disaster recovery 

Getting up and running quickly again after a disruption should never be underestimated. The longer the recovery takes, the greater the risk of permanent disruption. Among small companies that survive a disaster, 9 out of 10 fail within the following year if they’re unable to restore their operations within 5 days after the disaster (source FEMA).  

Often it can be the case that companies’ backup their data off-site but do not have the required computing equipment such as servers at another location to restore the data. This means that although the data is safe, it can still take a number of days/weeks to acquire the necessary hardware to restore the data and get systems back up and running. 

At CORE we are backing up customer data every 15minutes and transferring it directly to a standby system at a different datacentre. All the hardware required to run the system is already set up and ready to go when required. 

What are the key elements of a good disaster recovery plan 

1. Identify your key data  

Businesses need to first identify the data they require to operate their business. This could be customer data, application source code, infrastructure configuration. Once identified you then need to determine how frequently you must back up the data to ensure it’s still useable in the event of a disaster recovery event. 

For example, you may decide the source code servers only need backing up off-site once per day because the business could survive losing a days’ worth of development changes. But, a customer’s databases where there can be tens of thousands of updates per day may warrant a backup multiple times per hour.  

2. Backup your key data off-site 

Once businesses have identified the data they need to operate, they need to ensure the data is being backed up off-site at the required intervals. The majority of companies will be backing up the data, but often within the same datacentre or data could be backed-up online and could be accessed/deleted by a malicious actor. 

We backup our data to two offsite locations: 

• To a secondary datacentre where we can get the system back up and running very quickly. 

• Encrypted and stored on immutable storage in the cloud. This protects us from a malicious person destroying our data either via deletion or ransomware. 

3. Identify the key infrastructure required to run the application         

Backing up all of your key data, is the most important thing but only half of the solution. In the event of a disaster recovery situation, you need to get the infrastructure back online and accessible to your customers.  

Businesses need to decide whether they’ll purchase and run a secondary set of infrastructure at all times or acquire the infrastructure via a purchase or lease when they invoke the disaster recovery plan.  

Another option is to run the disaster recovery site from a cloud service provider such as AWS or Azure but this requires the business to have specialist in-house knowledge to run the application from such a provider.  

At CORE we are running production equivalent infrastructure at a secondary site at all times in case we need to failover to disaster recovery. This allows us the ability to get the system back up and running very quickly since we have all of the data + infrastructure. 

4. Disaster recovery rehearsals  

Businesses need to be well rehearsed on implementing their disaster recovery strategy, identify all key personnel within the business required and running through as many steps as possible to bring the disaster recovery site online.  

Ideally this would happen at least every 12 months and involve customers where possible. 

At CORE we run a full rehearsal every 12 months, and additionally test key aspects of the plan monthly. These monthly checks include spot checks on data and powering on virtual machines.  

5. Monitoring and review 

Once your disaster recovery plan is set up and well tested there needs to be a system in place to monitor the backups and processes required to support the strategy and check it is running as planned. You don’t want to failover to your disaster recovery site to then find out the database backups have been failing for the past 3 months! 

At CORE we have enterprise monitoring systems in place, which includes checks on our disaster recovery process. These checks are running every minute and alerts are sent to our systems administration team. Additionally, we are running monthly checks on the disaster recovery systems. 

Importantly, the plan should be reviewed every 12 months or when any significant changes are made to the system. 

Getting the protection you need for your business and ensuring disaster doesn’t lead to disruption should be a top boardroom priority. Choosing a technology provider who understands this is crucial – your data is in their hands so ask in detail about disaster recovery plans when appointing your team. Read more about CORE’s services