Cloud Infrastructure

Shorter post today, was out on my bike this morning, up to Skyline and down a couple of times. By the way, as I rode down past the redwoods on one tiny apparently-deserted lane, a guy stopped me and had me take a picture of him and his wife alongside their new Ferrari, using the Santa Cruz mountains as backdrop. Very Silicon Valley…

Anyhow, if I may quote myself, back on Jan 6th, I had a tweet:

Duncan Greatwood Web bad lately? #Skype #Hotmail #Twitter #LinkedIn (today!)#Google -calendar flaking. Can we fix cloud? http://bit.ly/ggnbzv #perfectstart via Twitter

I might have included Tumblr, which is not working again today. Or Twitter itself – the Tweet above could not be pulled directly from Twitter due to overload, I had to pull it from LinkedIn…

What’s wrong with cloud/web infrastructure?

The main issue is that running a truly large infrastructure involves a lot of layers, a lot of complexity, a lot of custom work pasting things together, a lot of less-than-100%-perfect components (software, load-balancers, hardware…), and a lot of diving-catch human intervention.

One requirement to get it fixed is a standardized, integrated, automated solution.

Amazon announced their “Elastic Beanstalk” service today, which is an attempt to package together Amazon Web Services in a more pre-baked way.

OpenStack and Eucalyptus are both examples of attempts to build a pre-packaged web infrastructure.

The “DevOps” movement – is it big enough to be called a movement yet? – also fits here, with their attempt to break down the barriers between application software development and cloud infrastructure management.

Although some people, such as those mentioned above, have made a start, there’s a huge amount still to do here. The width and depth of the stack is much greater than anyone has tried to tackle yet. The stack would defacto reach into core application software development libraries and methodologies (which is in part motivating VMWare’s push to developers).

As a startup area, it has some challenges.

Almost by definition, the integrated stack market will in part be a market for comprehensiveness – whereas small companies often find it easiest in markets that benefit from focus.

Second challenge – the really big cloud operators have their own stacks, often messy and the result of historical accident, but nonetheless hard to displace.

A startup in the space might to do best to target enterprise clouds, at least initially.

Another tactic would be to go after the as-yet-unintegrated parts of the stack, to be complimentary to OpenStack, VMWare, or Eucalyptus, in an attempt to narrow the initial scope of the problem. If done right, this might have the double benefit of making it applicable sooner to the large-scale cloud companies – Facebook, LinkedIn, Twitter and all the rest – if it tackles the automation of something they haven’t well-automated already themselves.

When I have some time for a little more in-depth write-up, I’ll sketch out what “the stack” consists of.

[Hat-tip: There’s a nice stream of comments from commenter “M K” on LinkedIn against my original Tweet, for those who are connected to me on LinkedIn]