The Triage Manual — Technical Guides for IT Emergencies

Failed Domain Controller — recovery without making it worse

Sat, 30 May 2026 06:45:20 GMT

A DC has failed (hardware, OS, NTDS corruption, or network isolation). The danger isn't the failure — it's the recovery shortcut that breaks the rest of the forest.

Seizing FSMO roles from a dead Domain Controller

Sat, 30 May 2026 06:45:20 GMT

Original FSMO holder is unrecoverable. Seize roles to restore directory operations — and ensure the original DC never returns.

Kerberos RC4 Hardening Phase 2 (April 2026) Breaking Authentication — Service Accounts and Legacy Applications

Sat, 30 May 2026 06:45:20 GMT

Microsoft's April 2026 Kerberos RC4 hardening Phase 2 enforces AES-only Kerberos ticket encryption on domain-joined Windows clients and Server Domain Controllers, rejecting RC4-HMAC by default. Service accounts, computer accounts, and legacy applications whose msDS-SupportedEncryptionTypes attribute does not include AES128 (value 8) or AES256 (value 16) — or which hard-code RC4-HMAC negotiation — will fail Kerberos authentication after the April 2026 cumulative updates are applied. Resolution r…

AD Replication Failure between DCs

Sat, 30 May 2026 06:45:20 GMT

Domain controllers are not replicating, or a decommissioned DC has left stale metadata in Active Directory. Changes made on one DC do not appear on others; stale DC objects cause replication errors, DNS pollution, and KCC failures. If left unresolved, the directory diverges and tombstone lifetime becomes a hard deadline.

Group Policy not applying

Sat, 30 May 2026 06:45:20 GMT

Policy changes reach the domain but never appear on clients — or apply inconsistently. Diagnose: link, scope, filter, processing, replication.

SYSVOL / DFSR replication failure

Sat, 30 May 2026 06:45:20 GMT

SYSVOL is not replicating between DCs — Group Policy and login scripts diverge across the domain. DFSR replaced FRS years ago; recovery still goes wrong regularly.

Enumerating AD Security Group Members (Including Nested Groups) Without Domain Admin Rights

Sat, 30 May 2026 06:45:20 GMT

A domain user on Windows 8 or later needs to view both direct members and nested groups within an Active Directory Security Group but lacks domain admin privileges and access to standard AD management tools. The built-in 'net group /domain' command is insufficient as it omits nested groups, and dsget may not be installed. The primary resolution is to use the native 'Search Active Directory' feature accessible through Windows Explorer's Network view, which exposes full group membership including…

GPO Fails to Apply — Broken Netlogon Secure Channel (Machine Account Password Desync) on Windows 10/11

Sat, 30 May 2026 06:45:20 GMT

Domain-joined Windows 10/11 workstations stop applying computer and user Group Policy Objects when the Netlogon secure channel between the local machine account password and Active Directory falls out of sync. The desync typically occurs when the workstation is offline during scheduled machine account password roll cycles, causing the DC to reject the workstation's authentication. gpupdate /force returns network connectivity or access denied errors; Test-ComputerSecureChannel returns False. Rem…

Identify Which Domain Controller Is Authenticating the Current User Session

Sat, 30 May 2026 06:45:20 GMT

When troubleshooting Group Policy application, authentication failures, or AD replication issues, it is often necessary to determine which Domain Controller handled a user's logon session. The built-in Windows environment variable %LOGONSERVER% provides this information instantly from any Command Prompt without requiring administrative privileges. This entry covers how to retrieve, interpret, and act on that information.

Listing AD group memberships for a user with 'net user /domain'

Sat, 30 May 2026 06:45:20 GMT

The 'net user' command queries the local SAM by default and does not accept 'DOMAIN\user' syntax, so attempts to enumerate Active Directory group memberships fail. Appending the '/domain' switch with the bare sAMAccountName redirects the query to the current logged-on domain controller and returns the user's global and local group memberships.

Domain Controller Misclassifies Network as Public at Startup Due to NLA Race Condition

Sat, 30 May 2026 06:45:20 GMT

A Windows Server 2008 R2 Domain Controller incorrectly identifies its network connection as a Public Network at startup, causing Windows Firewall to apply the Public profile instead of the Domain profile. This occurs because Network Location Awareness (NLA) evaluates the network type before AD Domain Services is fully initialised, or because the default gateway is missing or unreachable via ICMP at boot time. The issue resolves transiently by disabling and re-enabling the NIC, but the permanent…

Domain Controller CPU Spikes Caused by Full Security Event Log

Sat, 30 May 2026 06:45:20 GMT

Windows Server 2008 Domain Controllers hosted on vSphere exhibited cyclic 80–100% CPU spikes every 2–3 seconds traced to the EventLog service (svchost.exe). The root cause was the Security Event Log reaching its configured maximum size of 131,072 KB, forcing the EventLog service into continuous and expensive in-place overwrite operations. Clearing the Security Event Log and adjusting the retention policy immediately returned CPU usage to approximately 5%.

DNS Dynamic Update Failures — Event ID 4015 on Active Directory-Integrated DNS Zones (DomainDnsZones / ForestDnsZones)

Sat, 30 May 2026 06:45:20 GMT

Active Directory-integrated DNS zones on Windows Server 2016/2019/2022 Domain Controllers refuse dynamic updates from clients and DHCP nodes, logging Event ID 4015 ('directory service threw a critical error') in the DNS Server event log. The root cause is replication latency, lockups, or stale dead replica references within the DomainDnsZones or ForestDnsZones Application Directory Partition, preventing the DNS Server service from writing or replicating zone data. Immediate remediation is a seq…

GPP Printer Deployment Fails with 0x80070bcb After KB3170455 (MS16-087)

Sat, 30 May 2026 06:45:20 GMT

Windows 10 clients fail to install printer drivers deployed via Group Policy Preferences with error 0x80070bcb after security update KB3170455 (MS16-087) is applied. The update blocks silent installation of non-packaged, unsigned printer drivers through Point and Print, rendering GPP-based printer deployment ineffective regardless of Point and Print Restriction policy configuration. Resolution requires sourcing a packaged, signed driver from the manufacturer or adopting an alternative deploymen…

SQL Server Kerberos Falls Back to NTLM Due to Duplicate SPN After Server Rename or Migration

Sat, 30 May 2026 06:45:20 GMT

Following a SQL Server migration where the new server was renamed to match the old one, Kerberos authentication silently falls back to NTLM because a conflicting MSSQLSvc SPN remains registered under the old domain service account. The duplicate SPN causes Active Directory to be unable to uniquely resolve the Kerberos principal, forcing NTLM. Removing the stale SPN from the old account and allowing AD replication to propagate restores Kerberos authentication.

Force Domain Controller to Re-register AD DNS Records Without Restarting Netlogon

Sat, 30 May 2026 06:45:20 GMT

When a Domain Controller fails to register or maintain its AD-specific DNS records (SRV, CNAME, A) in DNS zones such as _msdcs, _tcp, _udp, and _sites, clients lose the ability to locate domain controllers. Running 'ipconfig /registerdns' is insufficient as it only handles A and PTR records; the correct fix is 'nltest /DSREGDNS' (optionally with '/SERVER:' for remote execution), which immediately triggers netlogon to re-register all AD DNS records without requiring a service restart. If n…

No Domain Controller Authenticating Users — Total AD Authentication Outage

Sat, 30 May 2026 06:45:20 GMT

When no Domain Controller in the environment can authenticate or authorize users, all domain-joined logins fail with 'No logon servers available' errors, and AD-dependent applications cease to function. Root causes span DC unavailability, DNS resolution failure, Netlogon service outage, Kerberos time skew, or firewall port blockage. Resolution requires systematically restoring DC reachability, DNS SRV record resolution, Netlogon service health, time synchronization, and firewall access to AD po…

dMSA Ouroboros: Self-Sustaining Credential Extraction via Delegated Managed Service Accounts in Windows Server 2025

Sat, 30 May 2026 06:45:20 GMT

Windows Server 2025 domains are vulnerable to a credential extraction technique called 'dMSA Ouroboros' that abuses delegated Managed Service Account permissions to extract privileged credentials without Domain Admin rights. The attack exploits the dMSA credential migration mechanism by linking attacker-controlled dMSA objects to privileged accounts via msDS-ManagedAccountPrecededByLink attributes, causing the DC to provision those credentials to the attacker. The technique is self-sustaining a…

Windows Server Domain Controllers Enter Reboot Loop After April 2026 Cumulative Update

Sat, 30 May 2026 06:45:20 GMT

Windows Server domain controllers (2016, 2019, 2022, 2025) enter a continuous reboot loop after installing the April 2026 cumulative or security updates, rendering Active Directory services unavailable. Microsoft has acknowledged the issue and released out-of-band updates as the primary remediation path. Where out-of-band updates are not yet available or cannot be applied, resolution requires booting into Safe Mode or DSRM to uninstall the offending update, or restoring from a pre-patch backup.…

SMB File Shares Accessible via IP but Failing via FQDN — Kerberos SPN Mismatch or Duplicate Fault

Sat, 30 May 2026 06:45:20 GMT

Users cannot access SMB file shares using the server FQDN (e.g., \\fileserver.domain.local\share) and receive 'Access Denied' or 'Network path not found' errors, while direct IP access works normally. The root cause is enforced Kerberos authentication hardening that strictly rejects NTLM fallback when a Service Principal Name (SPN) mismatch or duplicate exists across Active Directory objects. Remediation requires auditing the domain for duplicate SPNs using setspn -X on a Domain Controller, the…

SMB File Shares Accessible via IP but Failing via FQDN — Kerberos SPN Mismatch or Duplicate Fault

Sat, 30 May 2026 06:45:20 GMT

Users receive 'Access Denied' or 'Network Path Not Found' when accessing SMB shares via FQDN (e.g., \\fileserver.domain.local\share) while direct IP access works normally. The root cause is Kerberos authentication hardening that rejects NTLM fallback when a Service Principal Name (SPN) mismatch or duplicate exists across AD objects. Resolution requires auditing for duplicate SPNs using setspn -X, removing stale registrations, and ensuring DNS CNAME aliases have corresponding HOST/ SPN attribute…

DNS Dynamic Update Failures — Event ID 4015 on AD-Integrated DNS Zones (Application Partition Replication Lockup)

Sat, 30 May 2026 06:45:20 GMT

Active Directory-integrated DNS zones refuse dynamic updates from clients and DHCP nodes, logging Event ID 4015 ('directory service threw a critical error') on Domain Controllers. The root cause is latency or replication lockups within the Application Directory Partition (DomainDnsZones or ForestDnsZones), often compounded by stale or dead DC replica references in the partition properties. Left unresolved, stale DNS records accumulate across the fleet causing broken connectivity and automation…

Exchange Online mail flow problems

Sat, 30 May 2026 06:45:20 GMT

Mail not delivering, NDRs, mass quarantine, or hybrid connector failure. The diagnosis runs from envelope sender to mailbox, in order.

Exchange 2013 mailbox database evacuation after eseutil offline repair

Sat, 30 May 2026 06:45:20 GMT

After eseutil.exe offline repair is performed on an Exchange 2013 mailbox database, Exchange emits hourly warnings that Exchange-level logical consistency can no longer be guaranteed and requires the database to be evacuated. The recommended remediation is to use New-MoveRequest to migrate all affected mailboxes to a healthy database, which is more reliable than PST export/import. Individual item-level corruption encountered during migration can be handled by increasing the BadItemLimit paramet…

Exchange On-Premises Federation Trust Broken with SOAP Auth Failure After Office 365 Domain Verification

Sat, 30 May 2026 06:45:20 GMT

After deleting and recreating an on-premises Exchange federation trust during Office 365 domain verification, Test-FederationTrust fails with SOAP fault errors 0x80048821 / 0x80041012 ('passwords do not match') when requesting a delegation token from the Microsoft Federation Gateway. Despite the misleading error wording, the issue is caused by a certificate or ApplicationIdentifier binding mismatch introduced by trust recreation, and may be compounded when the remote federated organisation has…

Exchange Online Mail Flow Disruption: onmicrosoft.com Outbound and Direct Send Changes

Sat, 30 May 2026 06:45:20 GMT

Microsoft's upcoming Exchange Online service changes will impact tenants sending outbound mail via the default onmicrosoft.com domain and those relying on Direct Send for inbound routing. The Change Optics Report, available in public preview within the Exchange Admin Center, proactively surfaces affected mail flows before enforcement occurs. Administrators must identify impacted senders, reconfigure outbound connectors to use verified custom domains, and migrate Direct Send sources to SMTP AUTH…

Exchange 2007 TLS Certificate UntrustedRoot Failure for Domain-Secured SMTP Delivery

Sat, 30 May 2026 06:45:20 GMT

Exchange 2007 fails to deliver mail to domain-secured recipient domains when the remote SMTP server presents a TLS certificate issued by a private or unknown certificate authority, resulting in an 'UntrustedRoot' validation error. Mail flow to the affected domain is blocked while delivery to non-domain-secured domains continues normally, and a temporary workaround exists by removing the domain from TLSSendDomainSecureList. Resolution requires identifying the remote server's certificate issuer v…

Exchange ActiveSync Certificate-Based Authentication Retirement — Mobile Email Migration

Sat, 30 May 2026 06:45:20 GMT

Microsoft is retiring direct certificate-based authentication (CBA) for Exchange ActiveSync (EAS) in Exchange Online by end of 2026, announced May 8, 2026. Organizations relying on CBA for mobile email access must migrate to Modern Authentication (OAuth 2.0) before the deadline or face complete mobile email connectivity failure. This entry covers auditing current CBA usage, selecting a replacement authentication method, and executing a staged MDM-driven migration.

Microsoft 365 SMTP Relay Configuration — Enabling IP-Based SMTP Sending for On-Premises Devices via Exchange Online Connector

Sat, 30 May 2026 06:45:20 GMT

On-premises devices (printers, scanners, applications) fail to relay outbound email through Exchange Online when no inbound connector exists to authenticate by IP address. Without proper configuration, SMTP submissions to smtp.office365.com are rejected because the device lacks a licensed mailbox credential. Resolution requires creating an inbound connector in Exchange Admin Center that allows relay from specific static public IP addresses, eliminating the need for per-device credentials.

Locked out by Conditional Access

Sat, 30 May 2026 06:45:20 GMT

A Conditional Access policy is blocking everyone — including admins. Recover via break-glass, then disable the offending policy, then investigate cause.

Azure AD Connect Export Error 8344: Insufficient Access Rights on AD Connector Account

Sat, 30 May 2026 06:45:20 GMT

During Azure AD Connect synchronization, export operations fail with error code 8344 ('Insufficient access rights to perform the operation') when the AD DS connector account (MSOL_ or ADSync account) lacks the necessary permissions on Active Directory OUs or objects. This typically occurs after permission changes, new OU creation, or misconfiguration during setup. Resolution involves identifying the affected connector account and using ADSyncConfig PowerShell cmdlets or the Azure AD Connect wiz…

Teams won't sign in / connect

Sat, 30 May 2026 06:45:20 GMT

Teams desktop client failing to authenticate or load — users blocked from meetings, chat, calls. Desktop and web behave differently and that helps diagnose.

Intune compliance / enrolment failure

Sat, 30 May 2026 06:45:20 GMT

Devices fail to enrol, drop out of compliance, or refuse company resource access. Intune diagnostics span Windows, Autopilot, OEM and Entra.

Granular File and Folder Restore from Microsoft 365 Backup (SharePoint/OneDrive)

Sat, 30 May 2026 06:45:20 GMT

Microsoft 365 Backup supports granular restore of individual files and folders from SharePoint and OneDrive as of late April 2026, allowing administrators to recover specific items from a backup snapshot without rolling back an entire site or account. The feature requires the SharePoint Backup Administrator role and applies to sites and accounts enrolled in Microsoft 365 Backup protection ($0.15 per GB/month). This article covers prerequisites, the restore procedure, and how to verify successfu…

Entra ID / Microsoft 365 Authentication Prompt Loop — WAM BrokerPlugin Corruption (Error 1001)

Sat, 30 May 2026 06:45:20 GMT

Users on Windows 10/11 Enterprise experience persistent authentication loops when launching Outlook, Teams, or Excel, with sign-in appearing to complete but tokens failing to cache, producing error code 1001. The root cause is corruption within the Web Account Manager (WAM) data folder for the Microsoft.AAD.BrokerPlugin app package, preventing secure caching of OAuth refresh tokens. Resolution requires inspecting the BrokerPlugin AppData folder for permission or visibility issues, then re-regis…

Entra ID / Microsoft 365 Authentication Prompt Loops — WAM BrokerPlugin Corruption (Error 1001)

Sat, 30 May 2026 06:45:20 GMT

Users on Windows 10/11 Enterprise experience persistent authentication loops when launching Outlook, Teams, or Excel, with sign-in attempts failing to cache credentials and generating error code 1001. The root cause is corruption of the Web Account Manager (WAM) data stored under the Microsoft.AAD.BrokerPlugin local package folder, preventing OAuth refresh tokens from being securely cached. Remediation involves re-registering the AAD BrokerPlugin app package via PowerShell for the affected user…

Hyper-V host crashed — recovering virtual machines

Sat, 30 May 2026 06:45:20 GMT

Host has crashed, restarted unexpectedly, or VMs are stuck in saved/paused-critical state. Goal: stabilise the host first, then recover guests in the right order.

VMware ESXi host disconnected or PSOD

Sat, 30 May 2026 06:45:20 GMT

ESXi host has disconnected from vCenter, gone unresponsive, or hit a Purple Screen of Death. Restore management, then guests.

RAID array degraded or failed

Sat, 30 May 2026 06:45:20 GMT

One or more disks have failed in a RAID set; array is degraded, rebuilding, or offline. Goal: avoid a second-disk failure during rebuild.

Storage performance has collapsed (latency spike)

Sat, 30 May 2026 06:45:20 GMT

Disk I/O latency has risen to the point that VMs, databases, or file services are unusable. Find the bottleneck before throwing hardware at it.

VMware vSphere BRICKSTORM Malware — VCSA and ESXi Hypervisor Hardening and Defense

Sat, 30 May 2026 06:45:20 GMT

BRICKSTORM malware establishes persistence at the VMware vSphere virtualization layer (VCSA Photon Linux and ESXi hypervisors) beneath guest OS visibility, where traditional EDR agents cannot detect it. The intrusion exploits weak identity design, lack of host-based configuration enforcement, and absent monitoring within the virtualization control plane — not product vulnerabilities. Defenders must harden the VCSA Photon Linux layer using the Mandiant vCenter Hardening Script, enforce strict ne…

Hyper-V Guest VMs Freezing or Restarting Unexpectedly — 2025 Emergency Update Required

Sat, 30 May 2026 06:45:20 GMT

A defect in Hyper-V or its guest integration components causes Windows 10, Windows 11, and Windows Server guest VMs to freeze or restart without warning, with elevated impact in Azure confidential VM configurations. Microsoft identified the root cause and released an emergency update in 2025. The primary resolution is applying the emergency patch to both Hyper-V hosts and affected guest VMs. Frozen VMs can be force-reset via PowerShell as an interim measure while patching is arranged.

Hyper-V Live Migration Fails at 80–90% Due to Kerberos Constrained Delegation Misconfiguration After Security Hardening

Sat, 30 May 2026 06:45:20 GMT

Live Migration between Hyper-V cluster nodes fails consistently at 80–90% completion with security context negotiation or access denied errors. Recent security hardening updates enforce stricter Kerberos constrained delegation requirements, breaking the Virtual Machine Migration Service authentication handshake. Resolution requires explicitly configuring Constrained Delegation on host computer objects in Active Directory to permit the Microsoft Virtual System Migration Service on both source an…

Hyper-V Live Migration Fails at 80–90% After Security Hardening — Kerberos Constrained Delegation Misconfiguration

Sat, 30 May 2026 06:45:20 GMT

Live Migration of VMs between Hyper-V cluster nodes fails consistently at 80–90% completion with Event ID 21502 or 22038 indicating authentication negotiation failures. Recent platform security hardening updates enforce stricter Kerberos constrained delegation validation, breaking the Virtual Machine Migration Service handshake between hosts. Resolution requires configuring explicit Constrained Delegation on host computer objects in Active Directory, permitting the Microsoft Virtual System Migr…

Dell Computer Cannot Boot into Windows – ePSA Diagnostics, Boot Repair & Recovery

Sat, 30 May 2026 06:45:20 GMT

A Dell computer fails to boot into Windows, presenting as a blank screen, boot loop, BSOD, or 'No Boot Device Found' error. The issue may originate from corrupted boot files, a failing storage device, incorrect BIOS boot order, or damaged Windows system files. Resolution follows a structured path: hard reset, BIOS/ePSA hardware validation, Windows Startup Repair, bootrec MBR/BCD reconstruction, chkdsk, and OS reset or drive replacement if hardware failure is confirmed.

SQL Server on Hyper-V: 'The wait operation timed out' after application inactivity

Sat, 30 May 2026 06:45:20 GMT

SQL Server 2012 hosted on a Hyper-V virtual machine intermittently throws 'The wait operation timed out' errors on the first database query following a period of application inactivity, with subsequent requests succeeding normally. The root cause is stale query statistics causing the optimizer to generate inefficient execution plans, which — combined with any auto-update statistics trigger on first access — pushes query execution beyond the connection timeout. Executing sp_updatestats to refres…

Windows Server Licensing & CALs — Core Packs, KMS/MAK Activation, Compliance

Sat, 30 May 2026 06:45:20 GMT

Windows Server 2016+ uses core-based server licensing (minimum 16 cores per server, sold in 2-packs). Client Access Licences (Device or User CALs) are required for every user or device accessing the server. Miscounts, KMS failures, or edition confusion generate compliance risk and activation errors.

RDS CALs Exhausted or Licensing Server Unreachable — Users Blocked from Remote Desktop

Sat, 30 May 2026 06:45:20 GMT

Remote Desktop Services stops issuing sessions when the RDS CAL pool is exhausted or the RD Licensing Server cannot be reached. After the 120-day grace period expires, users are disconnected immediately after login. Emergency steps: extend grace or redirect to a working licensing server; proper fix is purchasing and installing sufficient Per-User or Per-Device RDS CALs.

Windows Server High CPU / Memory / Disk — Performance Degradation Diagnosis

Sat, 30 May 2026 06:45:20 GMT

A Windows Server becomes sluggish, applications time out, or users report slowness. Systematic diagnosis using Resource Monitor, PerfMon counters, and process-level tools identifies whether the bottleneck is CPU saturation, memory pressure, kernel pool exhaustion, or I/O queue depth.

NTFS Permissions Broken — Access Denied After Migration, Inheritance Change or Ownership Loss

Sat, 30 May 2026 06:45:20 GMT

Users receive 'Access is denied' on file shares or local folders after a server migration, permission inheritance was disabled accidentally, or ownership has been transferred to an unknown SID from a previous domain. NTFS and share permissions are evaluated independently — access is the more restrictive intersection of both.