I’m still getting used to the important part vCenter SSO (Single Sign-On) is playing in vSphere 5.1. In my home lab I was switching domain controllers from W2k8 to Windows 2012. Transferred FSMO roles, integrated DNS, changed IP addresses for DNS on all servers and all seemed fine. My w2k8-dom01 server was demoted and removed. Few days later when trying to make a vCenter connection, I couldn’t logon anymore. As a good Windows Admin I of course first rebooted the vCenter VM but (as all real admins know) that seldom fixes the issue. Diving in the vCenter log files at “C:\ProgramData\VMware\VMware VirtualCenter\Logs” I found the following error:
2013-01-27T10:46:20.302+01:00 [04304 info ‘[SSO][SsoAdminFacadeImpl]’] [FindPersonUser] 2013-01-27T10:46:20.427+01:00 [04304 warning ‘Default’] Warning, existence of user “VANZANTEN\Administrator” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.427+01:00 [04304 error ‘Default’] The user account “VANZANTEN\Administrator” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored. 2013-01-27T10:46:20.427+01:00 [04304 error ‘Default’] [ACL] Adding unresolved permission for user “VANZANTEN\Administrator” 2013-01-27T10:46:20.427+01:00 [04304 info ‘[SSO]’] [UserDirectorySso] GetUserInfo(VANZANTEN\vSphereAdminGroup, true) 2013-01-27T10:46:20.427+01:00 [04304 info ‘[SSO][SsoAdminFacadeImpl]’] [FindGroup] 2013-01-27T10:46:20.536+01:00 [04304 warning ‘Default’] Warning, existence of group “VANZANTEN\vSphereAdminGroup” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.536+01:00 [04304 error ‘Default’] The group account “VANZANTEN\vSphereAdminGroup” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored.
That is when the lightbulb in my head went on (hey it was Sunday morning and I didn’t have my coffee yet). I figured the vCenter SSO (Single Sign-On) service was still trying to talk to the old domain controller for authentication, so I opened the administration page of SSO: https://<your SSO server>:9443/vsphere-client/ Login with: admin@system-domain (or root@system-domain for the vCenter Server Appliance). You did write down that password, did you? Go to: “Sign-On and Discovery” -> Configuration and look at the identity sources. There it is, the reference to the old domain controller. 
Change vCenter SSO domain controller
Changing it is easy, click “Edit identity source” and change the name of the new domain controller. Test your settings and you would expect to be done then. 
Unfortunately I received the following error: “The edit identity source operation failed for the entity with the following error message. Cannot connect to one or more of the provided external server URLs: w2k8-dom01.vanzanten.local:389”.
Strangely enough it is still pointing at the old domain controller. The easiest way to solve this was to just delete the entry and create a new one with the new domain controller in. Just to be sure I restarted the vCenter SSO (Single Sign-On) service first and then vCenter Server would start without any issues.
Update:
In the comments on this post, Jad (JM) wrote that you can also use the domain name when entering the primary server URL. In my case I would NOT enter: “ldap://w12-dc01.vanzanten.local” as my Primary URL but just “ldap://vanzanten.local”. vCenter SSO will then query the domain for the special domain controller DNS record and use this to find the domain controller to talk to. Btw, this doesn’t work if you already have entered two URLs and want to change them and make the second one empty. Think it is purely a GUI issue. When I empty the second URL, save it and then edit to check again, it keeps the old entry. When you delete the whole entry and create a new one that has an empty secondary URL, you’re fine. Thanks Jad!
It’s always hard to determine what static links/references there are when replacing Domain Controllers and it is easy to overlook some. Thanks for the info.
If you have web installed can you still with with that? I am having that orob after the change.
This is an avoidable error on VMware’s part. Instead of statically retaining the DC, they should be using DNS lookups to find the DCs in the environment.
Sheesh!
Well, that is what I thought too :-)
Wow, if it’s pointing directly to a DC instead of AD, then that would imply just downtime on the DC would break SSO. That’s an *awful* design on VMware’s part, adding in a dependency to a single server. Can anyone confirm that downtime on the configured DC will break SSO?
You can enter TWO domain controllers !!! But in my case I had shutdown both the w2008 domain controllers and was running on the two new W2012 domain controllers. Btw, I do think it would be better if you would just enter a domain name and let SSO figure out through DNS what the domain controllers are.
It helps when not driving and posting…sorry.. Can you still login with AD creds via the Web client? Mine stopped working after adding new DC’s and upgrading identity source (unless using admin@System-Domain
No, you have to login with admin@system-domain or root@system-domain when using the vCenter Appliance
Not a Vmware issue per se :) An easy way to avoid this issue is to point to the domain instead of a specific host. In your case, you can simply specify vanzanten.local:389.
If you dig in the DNS setup of an AD domain, you’ll notice an empty (that is, ) A record for each DC.
Should you remove or add a DC, SSO will pick it up quickly enough, depending on the TTL in your DNS.
If you prefer to keep some control over the process, you can simply create multiple A records with the same name (something like ldap.vanzanten.local) pointing to whichever DCs you want to use.
Gah. Sorry about the errant tags, didn’t think Disqus would do this.
Thank you for the comment. I tested it and worked :-) I updated my post to include your tip.
This works fine with a single AD site, but when you have sites separated by WAN links, using domain.local will resolve to any DNS server at any site. Using domain.local just uses DNS round robin and in no way is related to how sites are separated in active directory sites and services.
I believe I have this error, but the vsphere service wont even start anymore so I can’t get in to make this change. Any ideas?
I’ve also gotten into habit of connecting to 3389 (global catalog) instead of 389 for multi-site configurations since discovering issues in the way referrals are handled causing authentication timeouts
Life safer! Thank you! Vsphere did have a backup DC to point to, but that guy ate it sometime last year and I demoted the main DC this weekend and woke up to find none of my NetApp SMVI snapshots took place :-O
Again thank you!
Glad it helped :-)
Hi Gabrie, Great article thank you. I encountered a similar problem during a DR site, primary site outage and therefore primary AD servers down – unable to login to vCenter from DR site. So to me looks like same issue as you. However, in vCenter 5.5 no longer specify ‘hardcoded’ AD server in SSO configuration, but domain name with ‘active directory (integrated windows authentication) option. Has anyone else come across this before? How to get SSO to pickup DR AD servers if it losses access to primary site DCs? Does the placement of the FSMO roles have any bearing on this?
Thanks, this article save me a lot of trouble…
Justo dropped by to say thanks! Your article helped me on a lot! It literally saved my weekend! Thanks!
Issue: whan you change URL, and in old url (that you changed) you have record with old DC, that now not exist (it was deleted or off), vCentre tries to connect to thet old DC and u have that error with null.
Fixing: temporary create Alias record with old DC name to name of someone existing DC om you DNS server. It`s better to create record on server for which vCentre queries DNS requests. Make “lookup oldDC_name” on vCenter server to check that`s it work OK, it wust be reply with IP of other DC. Chage domain controller in vSphere client, “null” error not appear. Delete alias record from DNS
Removing the identity source will remove all permissions for users and groups for this source, so if you have a lot of permissions configured this might not be an attractive option.
I found it easier to change the ldap paths in the database when my secure connection failed:
UPDATE [RSA].[dbo].[IMS_CONFIG_VALUE]SET [VALUE]='ldap://server:389'
WHERE [VALUE]='ldaps://server:3269'
GO
Restart the SSO service afterwards.
Amen
I had the “The edit identity source operation failed for the entity with the following error message. Cannot connect to one or more of the provided external server URLs” error as well. And the old server was already demoted. I cheated my way out, by logging in to the vcenter server over ssh and adding the line [newip] [newhostname] [oldhostname] to /etc/hosts
After that, I could change the ldap url easily as the credentials were now checked against the new server.