Crisis Management

So flickr had a hiccup yesterday. Well, truth be told, it was a major problem on their side: the image caches ran amok and delivered the wrong pics - not a few of them a bit on the more adult oriented side (as a sidenote, this proves what we all knew anyway: The Internet Is All About Porn). To the emotional outcry from lotsa lotsa users came the fact that the problem was not resolved by restarting the flaky cache server(s) but instead resurfaced once again. So finally, after quite a few hours of downtime (and I bet beet red engineers working overtime to find the bug and fix it) the system is back up.

So that's the exposition, which just about gives you an idea of the dimension of this thingy. It didn't? Well, then let me summarize: It Was BIG. However, flickr not only took down their site but pointed to their blog - in which Eric Costello did keep the users informed (if only tersely, but this is better than just a few lame marketing lines stating that all is beautiful and the system is just being enhanced yaddayaddayadda). When it was apparent that flickr would solve the problem he sat down and wrote a decent explanation of the problem - in a way to satisfy both non-technical users and the somewhat tech-savvy ones. He explains the issue without emotional overtures nor does he play it down:

To be clear, we regard this as a serious problem, but it is something that goes away as soon as we restart the malfunctioning servers (tonight we found that the servers were going insane again shortly after restarting, but we have isolated the problem and believe we have a permanent fix).

And finally, he concludes with:

We shamefacedly apologize for the inconvenience and the scare. We understand that it probably seems very, very strange and we know that many people got the impression that their photos were lost forever. But they should all be back now, safe and sound. And everyone who works on Flickr's engineering and technical operations teams are working double time to ensure that it never happens again. Thanks for your understanding and patience!

Folks, this is one of the best pieces of crisis management I have ever seen! It states the problem; it states the solution; it takes the blame where necessary and it gives a promise to the future. Now, if we could set this as mandatory teaching for all companies worldwide I would feel so much better.