Garbage In / Garbage Out

This post is part of a series based on a presentation I did to the London VMware User Group on February 25th, 2010 about the reality of Enterprise scale internal cloud platforms. To find other posts in the series, just look for the tag “Bringing the Cloud Down to Earth”.

The notion of GIGO is of course much older than I am, but it’s one of those concepts that is timeless. In relation to Cloud, it’s more pertinent than ever. The marketing hype would have you believe that Cloud is a panacea, and many people hawking their wares artfully dodge the subject of your existing tools and processes. But ignore these at your own peril. The COO of the company I work for has a great quote, which goes something like “God made the earth in 6 days, because he started with a clean slate.”. The same is true of internal Cloud (or whatever you want to call it – I’m going to call it that for the sake of convenience) – you could probably nail down the platform code and functionality that you want to launch with in a few weeks, but making the requisite changes to existing processes and integrating with existing tools in your environment is what will take the lion’s share of time to address.

But the thing is, you don’t need to start a Cloud project right now in order to start thinking about and hopefully addressing some of these aspects immediately – but you will need to assemble a virtual team to tackle the challenges as they are likely many and multi-faceted, and you are not Rambo.

And truth be told, you would do well to start trying to address these things before your project gets funding, at least start planting the seeds in the right places. The last thing you want to is burn though hundreds of thousands or even millions of dollars/euros/pounds only to deliver a substandard product because you spent too much time and money trying to get people to agree to process changes. Never underestimate the ability of people to resist change, no matter how broken things may be. And delivering something half baked is only going to make you look bad.

Let’s look at some of the key areas of consideration.

Inside the VM
There is loads of information out there in the VMware community about scripting / automating all manner of aspects of the virtual infrastructure. But that’s only a very small part of the battle – what about the stuff that goes on inside the VM? That’s where the real work lies.

I have yet to work in an enterprise that didn’t have some kind of “post build checklist”, and I’ve worked at plenty of places whose build systems and supporting infrastructure systems had no API to speak of. Now I don’t know about you, but the illusion of self service and automation is not Cloud as far as I’m concerned. Allowing someone to request a VM via a website and doing nothing more than creating an empty VM shell and generating a ticket for someone to manually create a provision in your build infrastructure or install the OS and then manually perform “the post build checklist” is not Cloud. Here are some questions to get you thinking.

Can you automate the allocation of IP addresses and registration of DNS records today?
Can you automatically create and update CMDB records today?
Can you automatically initiate server builds?
Do you have a post build checklist?
Can you do the reverse of the above in order to decommission something?

Obviously the above is not exhaustive, but most importantly the Cloud doesn’t have to be the driver to automate these things now! If you can’t do any of these things today, you might want to put that Cloud project on hold and address these things first.

Fuzzy Process
Think about the things that are done in your environment today. How many of them rely on “experience” or unwritten “rules”? Often when you embark upon an automation project, these things come flying out of the woodwork. You may feel that cleaning up stuff like this is outside the remit of your project – the processes are not well defined and/or inconsistent, but none of that is your fault. The reality is, if you want to implement Cloud then it’s exactly this kind of stuff that you need to fix because without Cloud, the status quo could probably be maintained. In some situations you may come up against people who have intentionally kept things this way because they feel they wouldnt be as “valauble” if they didnt do that stuff. Which is why it really pays to have a great project manager, they are much better at dealing with the “soft” side of things – you _really_ don’t want to point an engineer at operations people with insecurities!

Again, here are some questions to get you thinking about the kind of stuff I call “fuzzy process”.

Can you programmatically determine what vCenter/Datacenter/Cluster/Datastore to place the next VM?
Can you programmatically determine the next available hostname to use for a VM?

Pricing is probably the most contentious issue you will come across in an enterprise, because in order to get the price of your Cloud offering down you have to first understand what actually makes up the price of infrastructure and services (support, backup, monitoring etc) in your environment. In large environments, the complexity of this approaches that of putting something into space.

In many Enterprises, IT is run as a cost center – that is, it can’t make a profit or a loss. The books need to balance each month or quarter. Now imagine having to do that for an organisation with tens of thousands of servers, thousands of IT staff, hundreds of bits of software, and tens of consultants and vendors. Not only that, but how do you actually put a value on the automation of the manual tasks that you plan to removed with Cloud? I’m not just talking about building VM’s, it’s all the extra stuff around governance, inventory management, reporting, capacity planning, billing etc. It’s an unenviable task to be sure, but an absolutely necessary one. And it’s not without danger – danger in the form of “incentivisation”.

Price incentivisation is often built on shaky ground and with a certain degree of myopia. What is “the right thing” to do today may not be the right thing to do tomorrow. But if you make it artificially cheap to do the right thing today, what happens when the new right thing comes along? And in making something artificially cheap, how do you actually know if it’s the right thing to do! Only the _actual_ cost will tell you whether something is the right thing to do or not. That cost of course should include things like power, space, carbon emissions and the fact that you can have a much higher ratio of servers to admins if those servers are virtual.

Your Cloud project should deliver enough value to make it obviously cheaper than what you do today. If it can’t, then you’re already doing things right or there’s no point in going ahead with the project. If the business doesn’t see the value in dollar terms, adoption rates will be poor and your work will go unappreciated.

Congratulations if you made it this far :). Hopefully I’ve shown you that if you have issues like the ones above, ignore them, and your “Cloud” implementation ends up nothing more than a virtual platform with a self service portal slapped on top, you’ll end up with as much garbage at the end as what you had in the beginning. In fact, you may actually make things worse. But if you do come across things that are un-automatable or would be very difficult to automate, don’t throw your hands up in the air and accept the status quo – challenge it! Which leads me to the next topic, “Challenge Convention”



2 Responses to “Garbage In / Garbage Out”

  1. bruce Says:

    People do need to start challenging the cloud talkers since they missed the chance when the VM salesmen and blade salesmen came around. We’re all heading back to the mainframe, whether in one physical unit or smattered across many, it’s just a matter of matching the metaphor.

    To talk specifically to your post, I’m not sure I follow the ‘things to watch’ mentality as much as I follow the ‘clean slate’ logic. That’s going to be a key to leveraging a cloud, did you start from scratch? With how flexible the idea is, scratch building will provide the best environment and can be grown from the seed into the mighty oak pretty easily.

    Oh, and instead of thinking of IT as a cost center, tell business to turn off all their computers. What does their profit look like now?

    • stu Says:

      Hahaha, I like that last line – I’ll definitely borrow that 😉

      The clean slate approach is of course optimal, but in my experience not a viable option and something we certainly couldn’t do. But by all means if you have the chance, go for it and thank your lucky stars!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: