Thursday, February 26, 2009

GMail outage...

A couple of days ago there was a GMail outage. Short summary, a combination of taking down a data center and a bug in the load distribution code leads to overloaded data center, and thus a bunch of people can't get to their mail.

A friend ribbed me about the readyness of Cloud Computing, on the back of this outage. My initial reaction was to defend Google on uptime stats alone. Now that Google have spread the love and admitted the cause, I'm going out on a limb and stating that very few (if any) corporate IT shops are prepared for dealing with entire data centers being taken off line for maintenance.

I mean, sure, there are corporates who have back up data centers, and even have machines in stand by DR mode. I'd be exceedingly surprised, however, if everything actually worked. Most DR tests I've seen were epic fails, and the fun thing is that DR readyness tests usually happen once a year, if at all.

If a corporate wants DR to work, it has to be a regular thing. Schedule DR cut over every friday night, and keep repeating until it works flawlessly. And then bring it back every month to make sure it keeps working flawlessly. Otherwise the business as usual is going to be in for a shock when things go pear shaped for real.

Tuesday, February 24, 2009

System integration nightmares

I'm currently poking my way through an enterprise service bus, figuring out how to make it do stuff. I've just successfully written my most complicated hello world program to date. It includes message formatters, message routers, two seperate processing pipelines, and it's processed across a SOAP interface generated from a generated WSDL file. Over engineering for the win, right?

The underlying metaphor embedded in the configuration language is one of a pipeline. Pipe line stages can have filters on both input and output, the output stage can select the next stage depending on attributes of the message, and the center of the stage, the component, does some form of transformation, all the while being isolated from the message transformation and routing happening around it.

The issue I'm having with this metaphor is that pipe lines are not turing complete. I need the ability to take a message in over a SOAP connection, do a bunch of steps (including writing to our local database as well as querying and updating a pair of remote ERP systems), and then return a result over that said same SOAP connection.

Now, in the simple examples, this return can be done, because an input endpoint can have metaphor breaking property of being inout as opposed to just in. What I'm scrabbling through the documentation attempting to understand is how i pass this SOAP connection through the chain so that I can return the result, even after a long trail of transforms. Not sure if it's going to work, or if I'm just going to have to cheat...

The most interesting point to me is that the pipeline metaphor is insufficient.

Tuesday, February 10, 2009


Dad told me a story, a long time ago. It goes something like this.

Ted, a manager is asked by the CEO to come to dinner to discuss the possibility of stepping up to divisional chief. So on the appointed day, Ted clocks off after a day of fighting fires, goes home, gets changed, and heads in for dinner with the CEO.

On the way in, there is a terrible accident in a tunnel, and Ted is stuck in instant gridlock. (This story dates from the days before mobile phones, go with me on this one.) So after fighting his way through the accident debris, he arrives at his dinner appointment only an hour late. After explaining why he was late, due to the tragic circumstances in the tunnel, Ted was promptly told by the CEO that he wouldn't be getting the Divisional Chief position.

At the time I thought the CEO was being overly harsh on Ted, given that the circumstances surrounding Ted being late were outside of Ted's control. It is only now, looking back, that I understand the CEO's point of view.

I'm increasingly seeing that people split into one of two pretty neatly divided categories. Those who do things, and those whom things happen to. The language used by the two groups are largely not interoperable, let alone expectations over behaviour.

Odd, but true.