Home | Cloud | Change Control in Cloud Services
Change Control in Cloud Services

Change Control in Cloud Services

0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 Filament.io 0 Flares ×

The post mortem summary on Microsoft’s most recent Azure outage is available and online here.  The content makes for interesting reading.  To summarise, the outage occurred as a result of an unanticipated code error but was exacerbated by the accelerated rollout made by an employee who hadn’t adequately assessed the risks of pushing the change out so quickly.  I imagine that poor chap probably no longer works for Microsoft.

So what conclusions can we draw from this disclosure?

  1. Human intervention (and error) will always be a factor. Just look at the airline industry for direction there; a large percentage of aircraft crashes are based on pilot error.  However pilots are heavily trained and go through simulation practice to mitigate the risk they will make mistakes.  I wonder if Microsoft and the other cloud providers can say the same?
  2. More control is being taken away from humans.  The Azure environment is being changed to prevent this kind of rollout in the future and to enforce more conservative policies for deployments.  On the one hand this seems like a positive, but what happens when humans need to override this level of control?  If another scenario occurs that requires the accelerates deployment of a change, could that be done easily?

The Architect’s View

It’s good to see that Microsoft does appear to have sensible pre-testing of changes before they are deployed into the wider environment.  However human error will always be a factor and even more so when the risk exposure is so large.  This necessitates cloud providers employ the best people (not the cheapest), who understand environments in detail and get the big picture, not just their small subsection.  As features are added to cloud services, finding the right people is going to get increasingly harder.

Related Links

 

Comments are always welcome; please read our Comments Policy first.  If you have any related links of interest, please feel free to add them as a comment for consideration.  

Subscribe to the newsletter! – simply follow this link and enter your basic details (email addresses not shared with any other site).

Copyright (c) 2009-2015 – Chris M Evans, first published on http://blog.architecting.it, do not reproduce without permission.

About Chris M Evans

Chris M Evans has worked in the technology industry since 1987, starting as a systems programmer on the IBM mainframe platform, while retaining an interest in storage. After working abroad, he co-founded an Internet-based music distribution company during the .com era, returning to consultancy in the new millennium. In 2009 Chris co-founded Langton Blue Ltd (www.langtonblue.com), a boutique consultancy firm focused on delivering business benefit through efficient technology deployments. Chris writes a popular blog at http://blog.architecting.it, attends many conferences and invitation-only events and can be found providing regular industry contributions through Twitter (@chrismevans) and other social media outlets.
0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 Filament.io 0 Flares ×