After a TrueNAS ISO reinstall — the two things that broke that no one warned me about

The TrueNAS SCALE machine that runs my SMB shares was in trouble. Windows clients had stopped seeing the shares reliably — intermittent at first, then completely. When I dropped to the shell to check, systemctl status smb returned something worse than a failed service: it returned a service that no longer existed. Same for nmb. Same for winbind. The service wrappers were gone. Samba's binaries were still there. The actual smbd and nmbd executables under /usr/sbin were untouched. But the systemd units that normally wrap them, and the TrueNAS middleware that manages both, had lost too much of themselves to be repaired by unit regeneration or middleware restart. Something at the service-layer wiring had broken beyond in-place fix. The recovery answer, once I stopped trying to patch it, was a fresh ISO reinstall and a pool reimport. That got the service layer back. What it didn't do — what nobody had warned me it wouldn't do — was fix Windows authentication or fix ACL permissions on the imported datasets. Two things I'd inherited from the reinstall broke SMB access in ways that had nothing to do with SMB itself. This piece is the story of those two things. The SID mismatch that made Windows reject correct credentials. The missing default ACL entries that blocked recursive permission repair. And the fix sequence that restored SMB access to what it had been before the service layer disappeared. Same box as the ZFS suspended pool piece, different failure mode. When the storage layer goes wrong, ZFS suspends the pool and tells you exactly that. When the service layer goes wrong, the layer just disappears and Windows quietly stops answering. // why the fresh ISO was the answer The service layer on TrueNAS SCALE isn't just Samba. It's the systemd units that manage Samba. The middleware plugins that TrueNAS uses to expose SMB configuration through its GUI. The internal API endpoints that the middleware calls when you edit a share in the interface. All three layers were compromised. I tried the standard recovery moves first. Regenerating service units through the middleware. Reloading systemd. Restarting middlewared to force the internal API to reinitialise. None of it took. The problem wasn't a single missing piece — it was structural damage across the middleware, systemd, and Samba integration layers that had drifted enough that regeneration couldn't reconstruct working state from what remained. That's the moment where the answer stops being surgery and starts being a reinstall. Not the emotional decision — the reasonable one. When a service layer's structural integrity is gone, patching pieces individually just produces a service layer that limps. A fresh ISO reinstalls all three layers cleanly, at their intended versions, in their intended configuration. Fifteen minutes of install, a pool reimport, and the service layer is back to what it should be. That's what happened. Fresh ISO. Reimport of pools. smb.service, nmb.service, winbind.service all back. midclt smb.status returning cleanly. The GUI managing SMB again without errors. Then the actual story of this piece started. // the two things that break after reinstall With the service layer working again, Windows still couldn't authenticate to the shares. Correct credentials, rejected by every share on every dataset. The problem wasn't SMB anymore. The SID mismatch. When TrueNAS SCALE reinstalls, every user account has to be recreated. Same usernames, same passwords, same group memberships — nothing prevents you from recreating them exactly. But Windows doesn't care about your username. What Windows cares about is the SID. A SID is the Security Identifier that Windows uses to actually attach permissions to identities. Usernames are human-readable labels layered on top; the identifier that gets written into ACL entries is the SID. When TrueNAS creates a user, it generates a SID. When TrueNAS is reinstalled and the same user is recreated, a new SID gets generated. Not the same one. The SID is not derived from the username. Existing ACL entries on the datasets — the ones from before the reinstall — reference the old SID. Windows looks at those ACL entries, matches its authentication attempt against the SID recorded there, sees no match against the new user's new SID, and rejects the authentication with "Credentials do not match." The credentials do match. The SIDs don't. Windows is telling the truth in a way that doesn't help you fix it. The missing default ACL entries. The second problem shows up when you try to fix the first one by applying corrected permissions recursively. TrueNAS SCALE's ACL editor supports recursive apply — set permissions on a parent dataset, tell it to apply down through everything below, and it walks the tree. That's the normal repair for permission drift. Recursive apply on imported datasets errors out immediately: Error: dacl -- Default ACL entries are required in order to apply ACL recursively That message is opaque unless you know what a default ACL entry is. Access ACL entries govern the files and directories that exist right now. Default ACL entries govern what permissions get inherited by new files and directories created inside a parent. On a healthy TrueNAS dataset both are set. On a dataset imported from a pool after reinstall, only the access entries survived the import — the default entries didn't come across. Without default entries, recursive apply doesn't know what to inherit. It refuses to write any of the changes rather than apply them incompletely. // the fix sequence Once both problems were understood, the recovery ran in a specific order. Step one: establish default ACL entries. Recursive apply had to work before anything else was possible. Establishing default ACL entries on the affected datasets was the unblock. TrueNAS SCALE ships preset ACL configurations that include both access and default entries. The RESTRICTED preset is the conservative choice — root and the dataset owner get full control, everything else is denied by default. Applying it to the dataset writes both access entries and default entries in the correct form, then recursive apply works. Step two: recreate the SMB user with the intended group memberships. The SMB user recreation was straightforward through the TrueNAS GUI. Same username, same password, same primary group. The new user got its new SID assigned automatically. Nothing about this step required anything unusual — the account creation is standard TrueNAS SCALE user management. The systemd and middleware refresh sequence run after the reinstall to confirm the service layer was clean: midclt call service.generate_all_units systemctl daemon-reload systemctl restart middlewared Those weren't necessary for the ACL fix — the fresh ISO had already given us a clean service layer — but they're the standard TrueNAS SCALE service-layer refresh sequence, and running them confirmed no residual issues before the ACL work started. Step three: apply corrected permissions recursively. With default ACL entries in place from step one, and the SMB user existing with a new SID, the corrected ACL now needed to be written down through the dataset tree. The RESTRICTED preset had already established the ACL structure; adding the SMB user with full control on the dataset and applying recursively wrote the correct SID into every ACL entry through the tree. Step four: verify from Windows. Windows client connected to a share, authenticated with the SMB user's credentials, got in cleanly. No more "Credentials do not match." The credentials the user was typing had always been correct — they now matched an ACL entry whose SID pointed at the user actually trying to authenticate. The ACL work took the same session as the reinstall itself. Most of the time was working through the preset and recursive apply through several affected datasets. The recursive apply itself was fast once it stopped erroring. // what I'd make instinctive Three things went into the runbook after this. Snapshot the system config before major changes. TrueNAS SCALE lets you export the entire system config to a downloadable file — settings, users, ACL definitions, share configurations. Before this incident, I hadn't done it. If I'd had a config snapshot from before the SMB service layer failed, the fresh ISO reinstall would have restored not just the service layer but also the user accounts with their original SIDs. The SID mismatch problem wouldn't have existed. That single missing habit was the whole reason step two of the fix took the form it did. Config snapshots now go with every major change to the box. Pool imports don't recreate everything the datasets need. Default ACL entries are the visible case; they're the one that surfaces an error message. But the principle is broader. When you import a pool into a fresh TrueNAS installation, you get the pool structure and the actual file data — you don't automatically get all the metadata state that the original installation had built up around the datasets. Some of that state has to be rebuilt manually. Assume it, don't be surprised by it. Know the SID story before you need it. SIDs are how Windows actually attaches identities to permissions, and username is a display convention. Reinstalls change SIDs. Any ACL entry pointing at an old SID will silently fail to match a new user of the same name. The "Credentials do not match" error is misleading because the credentials aren't the problem. The identifier attached to the credentials is. None of these are theoretical. Every one of them was learned by doing the work slightly wrong the first time. // closing This piece is about the layer between storage and client — where user identity meets file permissions, where authentication decisions happen. When that layer breaks, the storage is still fine, the network is still fine, and the client is confused. The SMB failure category is different from the ZFS failure category. Different response, different recovery, different mental model. The next piece in this series is another storage-layer story — a ZFS pool where an entire RAIDZ1 vdev appeared to be gone, and the recovery required reading ZFS labels directly to confirm the disks were still valid pool members before touching anything. Different layer, different failure mode, different set of things that turned out not to be as broken as they looked. If you're running TrueNAS SCALE and haven't exported a system config recently, do it now. That's the single highest-leverage thing you can take from this piece. The config snapshot is what turns a fresh ISO reinstall from a story about SID mismatches into a story about ten minutes of downtime.