How to Effectively Manage an IT Crisis?

When I speak to other IT managers about some of the problems I have encountered, they look at me as if I am from another planet. How could one person have been involved with so many crises? Didn’t you ever ask yourself, David, if you were the common denominator? They could be right, of course. As one former boss said to me, “Just because you are paranoid, doesn’t mean people aren’t out to get you.”

The evidence is quite strong. I have been in three countries at the outbreak of their most serious civil disruption of the last 50 years. Not only that, but I been involved in the management in many major IT crises, three of which ended up on national news. What I would say to any manager who feels this couldn’t possibly happen to them is “Don’t be so sure.” Like any well-run IT organization, all the companies I have worked for had good, well planned and well-practiced crisis management plans.

The risk of a crisis or indeed any risk is the product of probability and impact. Reducing the likelihood (i.e. probability) of anything going wrong is important, but all the investment in the world can’t guarantee that you will avoid crisis. Many less-experienced managers focus so much on prevention, that they become highly vulnerable when things do go wrong.

One insight that all my crises have given me, is that it needs (at least) two things to go wrong at the same time to cause a crisis. Yes, if a server goes down, then presumably your well-run IT operation will have fail-over systems that will take over. It is only when the fail-over systems fail that you have a problem, or indeed a crisis. This happens more often than you might think.

To give you a quick story. When I worked in the Netherlands, our data centre had dual power supplies (i.e. separately fed) from the grid to our data centre, as well as a generator and a UPS back-up. So, when a mechanical digger went through the power cable, you might imagine that the second grid supply would take over. Well, it would have done if it wasn’t for the fact that the so-called independent supply was fed from a sub-station that was in fact fed by the primary supply. Still, at least we had the generator. Well, we would have done if finance had paid to have it filled up after someone siphoned the diesel into their car the week before. We were down to our UPS, which was also awaiting an upgrade. The number of systems it supported meant that we had only 30 minutes to shut everything down. As it turned out, the power came back on with less than 2 minutes to go, thereby averting what would have been a major recovery operation. If you think that this stream of negligence would never happen to you, let me ask you this question. It was one I prepared for a course I ran last week in Dubai:

Question: Which of the following organizations has not had a major data breach in the same 3-month period?

A US Securities and Exchange

B Equifax – Credit checking

C Deloitte - Consulting

D Blue Cross Blue Shield - Insurance

E Verizon - Telecoms

The answer is Blue Cross Blue Shield. Theirs was one month earlier. You see my point. Major crises are more common than most managers would ever believe. Given that a crisis will hit you sooner or later, what would I suggest? Three items spring to mind:

First, put together your top crisis management team which includes not only the key IT players (you almost certainly have this already), but also the key players from the rest of the business. Secondly, simulate different crises (e.g. fraud, hacking, fire) so you get the chance to practice together on a regular basis. And, thirdly, train your team with a structured problem management method, separating work-around solutions from root cause fixes.

About the Author

David McKean

David is a leading global trainer within IT management. He runs courses on IT management and leadership based on his well-received book publications Excellent IT Management and Excellent IT Leadership. The books are based on his own experience as an international CIO and those of the 2,000 delegates who have benefitted from his courses.

David presents at conferences worldwide as an authority on technology leadership. Specialist topics include IT strategy, change leadership, IT operational excellence, building world class IT and IT governance. He was one of the architects of the FastTrack business series with titles including FastTrack Strategy, FastTrack Innovation and FastTrack Project Management. The series has been translated into several languages and has sold over 50,000 copies in 20 countries.

David's most recent corporate role was as Head of IT for Cable & Wireless where he was responsible for a major technology transformation program, and the largest in-source project in the UK at the time. He has also worked as CIO for UPC in Holland and AT&T in Asia, and as Director of IT for Bouygues in France.

David has spent many years working in various IT leadership positions in different businesses. He holds an MA in Engineering Science (Electronics) from Cambridge University and an MSc in High Frequency Electronics from University College, London.