Custom specs keep re-running!

Host ESXi version: ESXi 5.1 Patch 4 (Build 1743533)

This was a scary issue we ran into . Custom specs keep re-running on a machine that was deployed 2 months ago.

Initially we were alerted because a domain admin kept getting locked out – one that was used to join this machine to the domain. Upon further inspection, we realized the machine was no longer on the domain. Eventually I checked the custom specs log at c:\windows\temp\vmware-imc\guestcust.log and I could see that the custom specs had been kicked off right around the time these weird issues started. I also found the following error in the log:

[2014-07-23T15:48:43 ERROR] Unable to set customization status in vmx.

That error lead me to this vmware article. Turns out we have the exact build # on our hosts. (keep in mind, vcenter and your hosts have different build #s). We followed the workaround outlined in the article and it solved our issue. To find out which machines of yours are affected, run some variation of this query in PowerCLI:

get-vm  | get-vmguest | where-object {$_.osfullname -like “Microsoft*”} | ForEach-Object {get-vm $_.vm | Get-AdvancedSetting -Name tools.deployPkg.fileName} | select entity,name,value

We are filtering by OS because we only manage the Windows boxes. Also, you can only query by OS if the VM is powered on as it relies on vm tools running. Any VMs that are returned with a value under ‘value’ need to be fixed. If ‘value’ is already null, then you are good to go. Not all machines even have this property… not sure why some do and some don’t.

Be aware, the article explains that custom specs will be kicked off if the machine is power cycled. We also found that if the machine migrates due to DRS it will kick off custom specs (this was our case). This article  also states that a snapshot could kick it off.

It is best to get this fixed ASAP if you are affected.

** EDIT ** – 9.26.14

I wrote a script located here that will notify us if any VMs are found that do NOT have the workaround applied. I set it up to run as a daily scheduled task.


One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *