Here’s the relevance of that little story: the final year of level 3 was pretty much learning how to prove or derive most of the level 2 concepts and formulas. What was the effect of that? By learning a smaller set of deeper knowledge, the higher level constructs became little more than logical conclusions. Instead of learning 500 formulas, I could just derive the 500 from a much smaller set of principles and formulas. So when it came time to do that final level 2 exam in the final year, I got a near perfect score despite completing the level 2 course a year earlier. I didn’t have to remember the what, because I knew the why. But it didn’t just stop there – my deeper knowledge of maths threw a whole new light on physics (excuse the pun) too.

The same happens with technology. The world of Cloud in some respects dictates we become detached from knowing what is behind the veil. We are told the hardware shouldn’t matter, neither should the hypervisor or even the operating system in the case of PaaS. And it shouldn’t, except for when it does. If you understand the fundamentals of virtualised hardware timers and process scheduling, you’ll be much more competent at anticipating which apps are likely to run into problems when running on an IaaS, and what you might recommend to prevent the problem from occurring in the first place.

There’s nothing wrong with only knowing the what, of course. But if you’re pwned by your curiosity anywhere near as much as I am, knowing what just isn’t enough.

Check it out here.

Anyway, the question was asked if it was possible to boot a VM from iSCSI directly without any involvement from VMFS. Having actually done this quite some time ago, I knew it was possible so I said “yes”, but unfortunately I couldn’t access the memory region that held the answer to the logical next question (how?), and I had to say “I’ll get back to you”.

The only way to do this on physical hardware without using chain loading (ie PXE booting some other program and loading gPXE via it) is to burn the gPXE ROM onto the NIC (I’m conveniently throwing gPXE boot via CD/floppy/USB into the chain loading bucket). Obviously this is not something you would want to entertain with VM’s. But luckily for us, VMware gives us the ability to actually specify a separate file to use for the NIC BIOS! So it’s really quite straightforward, here’s how to do it:

1. Go to the ROM-o-Matic website and create a boot rom with the following options:

2. You should now have a file named gpxe-1.0.1-8086100f.rom.

3. Create a VM with an e1000 NIC.

4. Upload the gPXE ROM file into the same directory as the .vmx file of the VM you just created.

5. Add the following line to the .vmx file:

e1000bios.filename = "gpxe-1.0.1-8086100f.rom"

6. You’re done – you can now take advantage of everything gPXE has to offer.

I tested this out on ESXi 4.1, and it still works like a treat. Why do you need to use an e1000? gPXE doesn’t currently have support for VMXNET3. Since it’s an open source project, perhaps someone from VMware could make that contribution?

This is a post I’ve been meaning to write for a while now, but for one reason or another I kept putting it off. So when Mike Laverick asked me about what subjects I wanted to cover in my Chinwag, I finally got around to talking about it in public. I say in public, because this is something I and many others been saying to VMware behind closed doors for a bloody long time now. And as Mike pointed out during the Chinwag, there’s certain sense of irony in applying this phrase to VMware, given that Paul Maritz is credited with inventing it (or at the very least popularising the saying in the IT world) over 20 years ago.

I’m not writing this post to be inflammatory. Nor do I feel the need to justify my comments any more than what was said in the Chinwag. I just feel that a little more clarity and elaboration is needed – when we started on the topic the conversation was skirting around several things at once, all of which were pretty negative. And although I wouldn’t go as far to label those first 20 or so minutes as “VMware bashing”, I can understand how it might be seen that way. So let’s get that straight – when I say VMware should eat their own dogfood, I mean it constructively. I’m not talking about stuff like vCloud Director 1.0 requiring a database from the least-VMware-friendly company on the planet, or that the vSphere Client isn’t supported as a ThinApp package. I’m talking about something much more fundamental than that. I’m talking about glass houses and throwing stones.

The focal point of my thoughts on this is the VMware management layer, specifically vCenter. A product that saw it’s first release way back in 2003. And from a 40,000 ft. view has changed very little since. Obviously vCenter 4.1 has many more features and is much more vertically scalable, but it’s essentially the same monolithic application that it was back in 2003. In some ways you could say it regressed with vSphere 4.0, because the previous largely stateless application server acquired a local LDAP service (i still don’t understand why that can’t be remote, but I digress…), invalidating the simplicity of the “warm standby” availability model you might have employed with VI3.

Over the past few years, the noise most people have been making around vCenter falls into 2 broad categories: scalability and availability. Early in life, VMware could afford to neglect vCenter availability and so could we – virtual infrastructures weren’t that large, and vCenter was nowhere near a critical component. If it went down, nobody really cared – the VM’s kept running. But fast forward a few years and the picture was rapidly changing – virtualisation was taking off, and expanding into new areas like VDI. And when VMware started going down the acquisition path in 2007 with the likes of Dunes and Propero, as well as developing their own products like SRM (which was in development for years before being released), it was obvious that vCenter was moving squarely into the critical path rather than lurking in the periphery of it. That was 3 years ago.

Now ordinarily, this course of events wouldn’t be enough to draw the ‘dogfood’ fire from me, but as we all know VMware has for a very long time been waging a war with developers and software companies. How many times have they urged us to have the fight with an application owner who said they couldn’t virtualise for some reason that boiled down to shitty application design / coding, or imaginary resource requirements? I can tell you in the 5 or so years I’ve been in the virtualisation game, I’ve had plenty of those fights. And at the end of the day, the argument invariably comes down to the path of least resistance – either the app needs to be upgraded / thrown out / re-written, or it stays the same and goes onto physical hardware. When you work for a company who’s primary source of revenue is not technology related, guess which one of those is the cheaper short-term option and thus wins most of the time?

VMware of course knows this very well, and to a certain degree the existence of these shitty apps makes VMware’s products all the more attractive. Features like vMotion, HA and Fault Tolerance are designed to provide a level of availability on the infrastructure layer that is unachievable in the application layer with most of the software found in the enterprise today. If all the applications we use had resiliency built into them, there would simply be no need for any of those infrastructure layer features, and a big part of the competitive edge VMware had for a long time might never have come into existence.

But VMware isn’t just any old business. It’s a software company. If anyone had the ability to implement architectural changes to an application, surely it would be someone young (in the software world) and bleeding edge like VMware? Apparently not. When we customers started howling about the lack of resiliency of vCenter, the solution we got was vCenter Heartbeat – a Band-Aid® to address what is, IMHO, a fundamental application design issue.

The fact that VMware chose to focus on the scalability aspects of vCenter first isn’t really the issue, it’s that they appear to have done so in isolation from the availability constraints. What’s the point of being able to manage hundreds of hosts and thousands of VM’s from a single application instance, if that instance is a single point of failure? Because of this lack of focus on availability, when vCenter moved into the critical path of our infrastructure, it fell into that poorly designed app bucket. Sure, it wasn’t a requirement when it was first developed, but it’s certainly been a requirement long enough for the problem to have been addressed at it’s root. And I don’t accept that VMware was so deaf or shortsighted that the availability requirement caught them completely off-guard.

I don’t know about your environments, but I’ll tell you how many other critical infrastructure applications (DNS, directory services, monitoring, backup, AV, etc) in my environment need to be protected with a technology like vCenter Heartbeat: 0. None. Nada. Zip. Zilch. Every single one of them can either run on a 3rd party clustering solution, or achieve high availability by way of a distributed architecture. vCenter on the other hand is monolithic and has not been certified to run on top of a 3rd Party clustering solution since at least version 2.5. A cynic might say that’s because they want to extract more money from customers by making them buy Heartbeat. Personally, I think it’s likely down to a lack of internal resources to QA such a configuration. Which is a pretty piss poor excuse – VMware isn’t any old software company, it’s a company that provides mission critical software. They should have QA resources coming out their ears.

But happily, there are changes afoot. Just look at vCloud Director – that’s how vCenter should have been a long time ago. At least nowadays my feedback to VMware is whole lot easier – I can just say “make vCenter like vCD”. We’ve been saying things to that effect to VMware for years now, and vCD gives every indication that they have been listening. (Aside: The reason I don’t mention that in the Chinwag is because vCD was released after the Chinwag was recorded and I wasn’t on the vCD beta).

There’s a saying that goes something like “God could build the Earth in 6 days because he started with a clean slate”. Whether you believe there is a God or not, there’s a lot of truth in that. And I certainly accept that changing the fundamental architecture of vCenter is not something that could happen overnight. But if VMware wants to go on the offensive against the current state of Enterprise applications it had damn well better make sure that it’s own backyard is in order first. I look forward to the day when all VMware management apps can be deployed to a PaaS and backed with a data fabric. A tcServer and GemFire based one of course .

I wanted to have a post ready to go to elaborate more on the whole VMware / dogfood bit, but as you’ve noticed I’ve been pretty busy lately (ie very quiet on the blogging front). And I guess it’s OK for me to say why… I am very proud to say I am the technical editor for the upcoming PowerCLI book from Al, Luc, Glenn, Arnim and Jon! And let me tell you, it ain’t easy when you’re dealing with masters like those guys.

Anyway, I hope you find the chinwag entertaining / interesting, and big thanks to Mr Laverick for asking me on. As I say at the end, if it wasn’t for Mike I might not have gotten far in this field and even this blog might not have gotten far – he was the first one to give me a shout out way back in the day, and my readership grew exponentially for a while after that. So again, Mike, thanks for everything.

Of course, the big announcement (for me anyway) was around the release of VMware vCloud Director. And while it looks completely awesome, the devil is in the implementation detail and I’ll be putting up some posts next week to talk about some of the not-so-shiny aspects of the product – someone’s gotta talk about the other side of the coin (why is it always me though!). But for now, I’m just gonna enjoy the Kool-Aid

While I’m here, I have been completely blown away at the number of people I have met, had really, really good conversations with, revealed my identity to, and had them come back with stuff like “NO WAY! You’re vinternals!?!?!” and then punched me in the face given me a big ol’ slap on the back. Seriously people, thank you all very much for the kind words. As a blogger, it’s really hard to tell how engaged your audience is – you can’t really tell that from “number of unique visitors” type metrics. So thanks for reading, and I’m really flattered that you guys appreciate what I have to say – it makes it all worthwhile.

The result is that powershell.exe starts chewing up memory like there’s no tomorrow… basically until your entire machine hangs. Lucky for me I don’t need to use PowerShell in anger on a daily basis, but this problem was enough for me to resort to running a vanilla XP VM whenever I needed to do stuff with PowerShell. But thankfully, someone took the initiative and looked into the problem with Microsoft, discovered the root cause, and a workaround. And so let it be known that I don’t get any credit for this post, but the person who should cannot be named for stupid corporate communications policy reasons.

And so here’s the workaround. Basically, you need to add the following to the powershell.exe.config file in the same directory as the powershell executable (both locations if you’re on an x64 machine – just create the file if it’s not there):

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <system.net>
  <defaultProxy>
   <proxy autoDetect="false"/>
  </defaultProxy>
 </system.net>
</configuration>


Et voila! Problem solved. Not sure if any other .NET apps are affected by this, but any PowerShell scripts that directly invoke System.Web.Client (for example) will be – I have a script to grab UUIDs via the iLO interface of HP boxes, and got the same problem whenever I invoked that script. But not anymore, hooray!

Saturday 28th
7pm onwards : vBeers San Francisco

Sunday 29th
7pm onwards : VMunderground Party

Monday 30th
4 – 7pm : VMworld Welcome Reception
9pm onwards : VMworld 2010 Tweetup

Tuesday 31st
5:30pm onwards (probably, but not necessarily in this order)
NetApp Virtualisation Community Party
EMC Customer Appreciation Party
Veeam Partay

Wednesday 1st
5:30 – 7:30pm : vExpert Party
7:30pm onwards : VMworld Party

Thursday 2nd
8pm onwards : v0dgeball (go vSquirrels! )

Really looking forward to meeting all the vFriends I have made over the years in person!

Even if the processes are documented and consistent, it usually results in a lot of complexity in the orchestration layer. This problematic not only because it can mean an excess of configuration data being stored in this layer (workflow tools should _not_ be used a configuration repositories!), but it also means the owners / developers of the orchestration layer need to gain a good understanding of the infrastructure component in question (a resource intensive process) and a loss of control by the operations team, who are in reality the experts!

The solution to this is to make sure that these backend components can truly operate _as a service_. That is, move all the component specific logic as close as possible to the component itself, and give ownership of the service endpoint back to the team who owns the service. An orchestration tool should be able to just call into the service endpoint with a set of requirements (Location: DC1, OS:Linux, RPO:Standard, RTO:Standard, DataVol:100GB) and have something underneath that endpoint handle all the capacity management elements and schedule configuration. In the world of today, this will likely mean some kind of custom development to build such a service endpoint. But that’s not such a bad thing – remember all that talk of “freeing people from mundane stuff so they can do more higher level stuff”? This is a perfect example.

Doing something like this is a good idea, Cloud or not. Service Oriented Infrastructure is one of those buzzwords that has been flying around for years, but the only work that seems to have been done is somewhat superficial (ie implementing SLA’s, service catalogs and the like). And the next time you see / hear a vendor start touting their wares as “cloud ready” or “built for the cloud”, ask them about this kind of functionality. And see how ready they really are.

I’m sure you’ve noticed things have been a bit quiet around here lately… no I didn’t get attacked by sharks on my holiday, but I did decide to take the plunge and turn my attention to learning Python in more depth. Actually, that’s just a small part of where I’m going – I’ve started to wean myself off Windows! That’s right, no more PowerShell, no more C#, no more Visual Studio… in fact if it all works out, the only thing I’ll need Windows for in the future is gaming.

Of course you can learn Python just as easily on Windows as you can any other platform, but I like to develop in the same environment I plan on deploying to. Which means getting familiar with a terminal based development environment, which basically means getting familiar with Vim. But alas, the standard Vim colour schemes just didn’t cut the mustard for me, and after trying a few on it came down to a choice between darkburn, and BusyBee.

In the end I just couldn’t decide between the 2, so I ended up with a bastardised mix of both which I have imaginatively titled BusierBee. For anyone learning their way around these kind of tweaks, this page is mighty handy to help you with terminal colours. Here is what it looks like:

There are obviously loads of Vim colour schemes out there, tell me about your favourite in the comments!