Last autumn I ran our SOC’s annual business continuity exercise. This involved decanting our entire operation to an alternate site at a facility provided by a dedicated business continuity supplier. Our contract guarantees short notice and sole occupier access to a large vanilla operations room filled with rows of desks and terminals. Within minutes of activating our gold disk, the machines were transformed from bland blank canvasses to exact working replicas of the machines in our SOC. At the flick of a switch analysts had immediate access to the full range of toolsets, data and intelligence that they have in our primary site. Less than 40 minutes after simulating denial of our SOC, we were 100% operational in a new location with the ability to deliver all of our services.
Even though the exercise went as expected, I still found it an impressive achievement. When things go well it is usually the result of a great deal of work, and this is no exception. Our successful exercise was only possible as a result of years of designing, fine tuning and maintaining a resilient architecture drawing on multiple datacentres and cloud solutions. The business continuity plan is planned in detail, and that plan is reviewed every 6 months to ensure every element continues to be fit for purpose.
The business continuity facility that we use is routinely manned by a solitary manager; for the most part it must be a terribly lonely job and he was clearly glad of the flurry of activity created by our exercise. Out of interest I asked him how frequently the facility is used for real in response to a genuine crisis. I didn’t expect it to be often, but the answer still surprised me. In the 5 years he had been working there, none of his hundreds of customers had ever activated their alternate site for a genuine crisis. Apparently, the vast majority of customers never even test activation, so for the most part he watches over an empty room, waiting for a crisis that, for the most part, never comes.
But this year a crisis did come, and it was a big one. The global coronavirus pandemic and the subsequent closure of offices meant that business continuity was suddenly front and centre in the minds of organisations of all shapes and sizes. Like everyone else we reached for our process documentation, dusted it down and set about putting our well-rehearsed plan into action. Except it didn’t work. Like virtually all business continuity plans it started with the assumption that our primary office space had suddenly become unavailable to us. It most certainly did not consider the possibility that all office spaces would be denied to us; our alternate site was unavailable for exactly the same reasons our primary site was unavailable.
Immediately we had to scrap the plan and set about writing a replacement based on dispersed local working. Initially this seemed like a daunting task, but it quickly became apparent that producing a workable plan was actually surprisingly easy. All of the design considerations we had made for our original plan meant that we had a fundamentally agile architecture that could be bent to accommodate our new requirements. The same applied to our processes; the fact that we had planned so rigorously meant that we understood what we needed to change in order to work in a novel and alternative way.
We began looking at this challenge two weeks before our offices formally closed, but within 48 hours we were very confident we could, for the first time ever, seamlessly move a large team that had always been office based to dispersed, remote working. We were right too, and when the day came that migration was seamless with no loss of service to our customers.
Crises are unpredictable by their very nature and, ultimately, they rarely happen in the way planned for. In the end our standing business continuity plan didn’t work, but the lessons we had learnt from our years of planning enabled us to design a workable alternative extremely quickly. Dwight D Eisenhower put it succinctly when he said Plans are worthless, but planning is everything.