ESXi, vCenter and Active Directory

I don't have any confirmation on this (in fact VMware support has no idea how this happened), but after spending 11 hours on Saturday rebuilding one of my vSphere servers (which houses the entire View environment), I think I figured out why 90% of my vm's went orphaned on me after I took the server offline for some hardware upgrades (like ironically, a fibre HBA to hook into our new EMC VNX5300).

When I originally built this vSphere environment, I did so with the thought that this would be all I get to work with. I didn't have any shared storage (not even a decent iSCSI/NFS store), nor did I have vMotion... and apparently I didn't really know exactly what I was doing ("gasp!" - getting that out was hard for me because I hate to think that I did anything wrong). I had an idea of what I wanted but VMware was still a little new to me. Fast forward six months, and I've gone through "Mastering vSphere 5" twice, read most of VMware's vSphere 5 documents and have taken the vSphere 5 class (my VCP test is registered for June 4th). I have a MUCH better understanding of how the whole vSphere environment works. So this is how I think I killed 96 virtual machines - causing me to re-register each and every one of them and taking down my entire View environment.

I added the ESXi host to vCenter using a domain account whose password changes every 60 days.

See, I went through and set up access to the ESXi host before adding it to vCenter. I really didn't understand how vCenter played such an important role in management until recently. Here's my train of thought (granted, none of this is verified - just my own suspicions):

• I add the ESXi host to vCenter with a domain account
• vCenter cache's the username and password used to connect and authenticate the ESXi host
• I add some VM's to the host through vCenter
• vCenter uses those cached credentials to register those VM's with the ESXi host itself
• My domain account's password is changed
• I then proceed to build out my entire view environment on this host
• Each time a VM is created in vCenter, vCenter adds it to it's database then tries to register it with the ESXi host using the original cached credentials
• uh-oh... those credentials are expired. vCenter gets to actually create the virtual machine files on the datastore but the ESXi host refuses to add the VM to its list of hosted virtual machines
• Fast forward to Saturday and I take the host down to put the new HBA cards in it
• When I bring the host back online, vCenter tries to re-add the host but is unable to do so using the cached credentials
• I have to re-add the host to vCenter, this time realizing I need to use the ESXi host root username and password.

Again, this is my speculation, but it seems to be the only logical answer to how this happened. I mean, there's no way taking a host offline on purpose should make 90+ VM's orphaned you know?

Anyways, if anyone ever reads this and they know of official documentation from VMware that supports this theory, I'd love to read it!