VMware Cloud Foundation 4.2 (management domain)
Here is a scenario that looks trivial but is not. You reset the root password on your management domain vCenter. Maybe it expired, maybe you were rotating it, maybe you just did not know the old one. The reset itself works fine. And then SDDC Manager quietly breaks.
The reason is simple once you see it. SDDC Manager keeps its own copy of the vCenter root password in its credential store. When you change the password directly on vCenter, you have changed it in one place only. SDDC Manager still holds the old value and keeps using it. This post is about the part nobody warns you about: what you have to do inside SDDC Manager to put the two back in sync, and the errors you will hit along the way if the GUI is not available to do it for you.
This is not a post about how to reset the vCenter root password. That part is well documented, and I will only summarise it. The value here is everything that happens after the reset.
The setup: what was done and what happened
For context, the starting point was a management domain vCenter whose root password was reset directly on the appliance. On a Photon-based appliance, you do this over SSH or the console:
passwd rootIf the appliance rejects your new password with Password has been already used, that is the Photon password history in /etc/security/opasswd. Clear it and try again:
echo "" > /etc/security/opasswd
passwd rootIt is worth understanding what that file is before you overwrite it, because doing things to files under /etc/security blind is how people break appliances. /etc/security/opasswd is the password history file. Every time you change a password, the PAM module pam_pwhistory stores a hash of the old one here, and by default it remembers the last five. On your next change it compares the new password against those five hashes and refuses anything that matches. That is the “already used” error.
Overwriting the file with an empty line wipes that stored history, so there is nothing left to compare against and the change goes through. You need this specifically when you are setting the password back to a value that is still in the recent history, which is exactly the case if you are resetting vCenter root back to the old password that SDDC Manager still has stored.
It is safe. opasswd holds only old hashes used for the reuse check. It is not the credential store. Your actual passwords live in /etc/shadow, which this does not touch. The only thing you lose is reuse prevention until the history rebuilds itself on the next few changes. Nothing about login or the running system is affected. Some Broadcom KBs write this as cat /dev/null > /etc/security/opasswd instead, which does the same job.
Then stop it expiring again, because an expired vCenter root password is its own separate headache:
chage -M -1 rootThat is the whole reset. Nothing exotic. The appliance is happy, you can SSH in with the new password, the VAMI accepts it. As far as vCenter is concerned, everything is correct.
The problem is that SDDC Manager was never told. It still has the old password stored, and it uses that stored copy to talk to vCenter. So the moment SDDC Manager next tries to authenticate to vCenter, it fails, using a password that no longer exists anywhere except its own database.
The symptom
The way this shows up is the SDDC Manager UI failing to initialize. It sits on the loading screen and never reaches the login page. If you go digging in the UI app log, you will find it retrying against the PSC and failing, then attempting to SSH into vCenter to pull SSO metadata and failing there too.
The important thing to understand is that SSO on vCenter is fine. The SAML metadata generation works. You can prove it by running the exact command SDDC Manager runs, directly on vCenter:
/opt/vmware/bin/sso-config.sh -get_sso_saml2_metadata /tmp/idpMetadata.xml && cat /tmp/idpMetadata.xmlIf that returns valid XML, and it will, then SSO is not your problem. The only thing wrong is that SDDC Manager is knocking on the door with the wrong key. It has the old password and vCenter has a new lock.
There is also a real risk here worth stating plainly. SDDC Manager retries on a loop. Every retry is a failed login against vCenter root with the wrong password. Enough of those and you can lock out the newly reset and otherwise valid vCenter root account. So before you start fixing, stop the retries. On SDDC Manager as root:
systemctl stop sddc-manager-ui-appThe fix: sync the new password into SDDC Manager by API
Normally you would do this in the SDDC Manager UI under Security, Password Management, using the Remediate option, which exists precisely for a password that was changed outside SDDC Manager. But the UI is down, which is the whole reason you are here. So we do it by API instead, following the raw manual method in Broadcom KB 305970. As with other SDDC Manager API operations, everything below runs directly against the appliance.
Take a snapshot of the SDDC Manager VM before you touch anything. This is a database write, and you want a way back.
Step 1: Find the vCenter and its stored credentials
Get the vCenter id from the platform database. On SDDC Manager as root:
psql -h localhost -U postgres -d platform -c \
"select vm_hostname,id,status from vcenter"Note the id and confirm the status is ACTIVE. If it shows ERROR, set it back to ACTIVE with an update on the same table before continuing. If you have queried this platform database before, for example when you had to clear a stuck lock on SDDC Manager, this will feel familiar.
Step 2: Generate an API token
Use your administrator@vsphere.local credentials:
TOKEN=$(curl -s -k \
-d '{"username":"administrator@vsphere.local","password":""}' \
-H "Content-Type: application/json" \
-X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
echo $TOKEN If echo $TOKEN prints a long token, you are good. If it prints null or nothing, the SSO login failed, and nothing below will work.
Step 3: Read the stored credential and get the credential id
This step matters more than it looks, and it is where most people, myself included, lose time. Pull the current stored vCenter root credential:
curl -s \
'http://localhost/credentials-service/credentials?entityType=VCENTER&credentialType=SSH' \
-H "Authorization: Bearer $TOKEN" | json_ppThe output looks like this:
[
{
"entityType" : "VCENTER",
"credentialType" : "SSH",
"entityId" : "a1b2c3d4-0000-0000-0000-vcenterid001",
"secret" : "VMware1!",
"username" : "root",
"id" : "e5f6a7b8-0000-0000-0000-credentialid1"
}
]Two things to see here. First, secret is the old password SDDC Manager is still using. That is your proof of the whole problem. Second, and this is the trap, there are two IDs in this record:
entityIdis the vCenter itself.idis the credential record.
The update in the next step must target the credential id, not the entityId. Broadcom’s own KB says this explicitly: use the id, not the entity id. If you PUT to the entityId you get a 404, wrapped in a generic runtime error that tells you nothing useful.
Step 4: Push the new password
Now update the credential with the password you actually set on vCenter, targeting the id from Step 3:
curl -X PUT \
'http://localhost/credentials-service/credentials/e5f6a7b8-0000-0000-0000-credentialid1' \
-d 'Homelab.24!!' \
-H "Content-type:application/json" \
-H "Authorization: Bearer $TOKEN" | json_ppA successful response echoes the record back with the new secret and a fresh modificationTime:
{
"secret" : "Homelab.24!!",
"credentialType" : "SSH",
"entityId" : "a1b2c3d4-0000-0000-0000-vcenterid001",
"modificationTime" : 1784415076405,
"id" : "e5f6a7b8-0000-0000-0000-credentialid1",
"username" : "root"
}Step 5: Restart services and bring the UI back
Restart the VCF service stack cleanly rather than poking individual units:
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.shGive it a few minutes, then load the UI in a fresh private browser window. With the stored credential now matching vCenter, the SSH into vCenter succeeds, SSO init completes, and the UI reaches login.
The traps that will cost you time
The procedure above reads clean. In practice, there are three specific things that will bite you, and none of them are in the tidy version of the KB.
The bang-bang password trap
If your password contains !!, and mine did, do not put it in double quotes. In bash, double quotes still allow history expansion, and !! expands to your previous command. So this:
-d "Homelab.24!!"silently becomes something like -d "Homelab.24echo $TOKEN", and SDDC Manager stores garbage. Use single quotes, which switch off all expansion:
-d 'Homelab.24!!'This is not an SDDC Manager problem at all. It is a shell problem. But it presents as a mysterious API failure, so it wastes your time in exactly the wrong place.
The allowed character list
SDDC Manager’s API is fussier about special characters than vCenter is. vCenter will happily accept a password that the SDDC API then refuses. The accepted set, per Broadcom:
- General: A-Z a-z 0-9
- Special:
! @ $ % ^ & ( ) - _ + ` ~ . < > ? - Not accepted:
# = { } [ \ ; : , / "
A . and a ! are both fine. But if you pick a vCenter password with, say, a # or a : in it, you will not be able to push it through the API, and you will have to reset vCenter again to something the API tolerates. Check the character set before you choose the new vCenter password, not after.
The endpoint that the KB gets wrong for some builds
This is the big one. Broadcom KB 305970 tells you to PUT to:
localhost/v1/system/credentials/On my 4.2 build, that path returned a 404, wrapped in a VCF_RUNTIME_ERROR with a reference token. Chasing the reference token through the logs showed the request had actually landed on LCM, on port 7400, which does not own that path:
grep -ri "" /var/log/vmware/vcf/
# ... points to a second token ...
grep -i "" /var/log/vmware/vcf/lcm/lcm.log
# status=404, path=/v1/system/credentials/ The path that actually worked was credentials-service/credentials/, which is the same service that answered the GET in Step 3. That is the lesson: if the documented PUT path 404s, use the exact service and path that answered your read, and target the credential id. Do not keep hammering the documented path assuming you got the syntax wrong. You did not. The path is wrong for your build.
Does this still apply to 5.x and 9?
A note on scope. I tested and ran everything in this post on VCF 4.2 only. I have not reproduced the manual procedure on 5.x or 9, and I am not going to claim it works identically there, because I have not verified it. What I can state below about newer versions is only what Broadcom’s own documentation says. Where I am assuming rather than confirming, I say so. I plan to test this on 5.x and 9 and will update the post when I do.
This is VCF 4.2, which is old, and yes, I will upgrade it. But before anyone dismisses this as legacy pain, it is worth separating what is documented from what I am guessing.
What is documented, and therefore fact: the desync behavior is not a 4.2 bug. Broadcom’s own current 9.x documentation states plainly that after you reset the password in a component, you must remediate it in VCF Operations. Remediation exists precisely because changing a password outside the platform leaves the stored copy stale, and that is true in 9 as much as in 4.2. So the fundamental gap, that VCF does not accept an out-of-band change silently and forces a manual resync step, is confirmed as still present in the newest release. That part I am not assuming.
What is documented about recovery, version by version:
- 4.x, what I ran here: no detection. Silent desync, the UI breaks, and recovery is the raw manual method in this post. A psql query against the platform database, a hand-built token, and a REST call whose documented endpoint can even 404.
- 5.x: recovery is documented to move into the GUI. Security, Password Management, Retry or Remediate, enter the new password. See Broadcom’s Remediate Passwords documentation.
- 9.x: documented as a real step change. VCF Operations detects the desync on a periodic check and raises a Disconnected alert pointing you at Password Management, where you click Remediate. 9.1 adds fleet-wide password policies with compliance checking. See Remediate Passwords in VCF Operations.
So the recovery experience is documented to have improved from silent break, to GUI retry, to proactive alert and one-click fix. Credit where it is due.
What I am NOT claiming
Here is the line I will not cross. I am not claiming that the exact manual, UI independent commands in this post work the same way on 5.x or 9. I have not tested that, and there is no Broadcom KB that says the 4.x manual method is identical on newer versions. The evidence actually points the other way. The 9.x recovery KB uses different steps, including a workflow query against the Operations Manager database rather than the platform database path I used here, and routes you through the Developer Center and Fleet Management UI. So assume the endpoints, the database queries, and the credential object shape may all differ on 5.x and 9. If you are on a newer version, use the documented remediate flow first, and treat this post as the 4.x reference and as background on why the desync happens at all.
But notice the assumption in the improvements
There is one point that holds regardless of version, and it is the important one. Every documented improvement above assumes the GUI is up. In the failure I actually hit, it was not, much like another VCF recovery scenario where SDDC Manager would not come up at all. The desync is what took the UI down. SDDC Manager could not initialize because it was trying to reach vCenter with the stale password, so it never reached login. The tool built to fix the problem was unreachable because of the problem.
When the UI is down, there is no Remediate button to click. That is true on 4.x by what I experienced, and by Broadcom’s own documentation it looks true on 9 as well, since even the current 9.x recovery KB routes remediation through the UI and therefore assumes you can open the console that shows the alert. A proactive Disconnected alert is worth nothing if you cannot load the page it appears on. I have not tested the UI down scenario on 9, so I state that as what the documentation implies, not as something I have reproduced.
The part Broadcom should own
So here is the honest takeaway, and it is more careful than a lazy “old version” dismissal. This is not purely legacy pain for two reasons. Plenty of customers still run 4.x in production today, and this is exactly what it looks like for them. And by Broadcom’s own documentation, the underlying design- that VCF forces a manual remediation after any out-of-band change- is still there in 9.
The documented improvements are real, but they all fix the easy case, the one where the platform is healthy and just needs to be told a new password. The one scenario where you most need a graceful fix, the desync that has taken the management UI down with it, is, at least on 4.2 where I saw it, still handled the raw manual way. Whether 9 offers a supported UI independent path for that specific case is something I want to test before I make any claim about it.
A platform that positions itself as full lifecycle credential management should have a supported, UI independent path to resync a single password, because the day you need it is the day the GUI is already down. Until I can confirm otherwise on a newer build, keep this procedure in your 4.x runbook. And note the small detail that even 9.x documentation still warns against characters like ! and # in the password to avoid parsing issues, so the character trap that cost me time here has not fully gone away either.
Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.
