A restored VMDK is corrupted almost every time because the backup image was already invalid before the restore started; the restore only reveals it. The three common root causes are Changed Block Tracking (CBT) silently returning incorrect data, a snapshot consolidation that was interrupted or blocked by a file lock, and underlying VMFS or storage-level damage unrelated to the backup software itself. Fixing the recurrence means identifying which of these three produced the bad backup, not just re-running the restore.
Key Takeaways
CBT can report the wrong changed blocks without raising any error, a documented defect affected vSphere 6.5/6.7, and a separate one affected ESXi 8.0 Update 2 after disk host-extend operations.
A backup job showing “Success” is not proof the backup is restorable; CBT invalidation and consolidation lock failures both corrupt data silently.
Corruption risk is concentrated around specific trigger events, disk resize, host crash mid-consolidation, storage firmware bugs, not spread evenly over time.
VMFS-level corruption (bad shutdown, HBA/RAID firmware faults, overwritten partition tables) produces the same symptom as a backup-software bug but requires a completely different fix (VOMA, not CBT reset).
The only reliable way to know a backup is good is to restore it and validate it, this is also what NIST’s Cybersecurity Framework 2.0 recovery guidance recommends for any backup and restoration program.
Incremental-forever backup strategies need a mandatory periodic full backup and a scheduled test-restore cadence specifically because the failure modes above are silent.
What Restores but Corrupted Actually Means
When a VMDK “comes back corrupted” after restore, it usually shows up as one of a few concrete symptoms: the guest OS fails to boot, the restored disk shows unreadable or garbage file-system data, checksums on restored files don’t match originals, or the VM powers on but applications inside it report database or file-system integrity errors. In every one of these cases, the corruption did not happen during the restore operation itself; restore is a copy operation. The data was already wrong in the backup repository, or the underlying source disk was already inconsistent when it was captured.
Why It Happens: The Root Causes
1. Changed Block Tracking (CBT) reports the wrong blocks
CBT is the VMware mechanism that tells backup software which blocks changed since the last backup, so incremental backups only need to copy those blocks. When CBT reports the wrong set of changed blocks, the backup software copies the wrong data, and every incremental backup taken afterward inherits the error, because each one is built on the previous, already-wrong copy.
This is not theoretical. Broadcom’s VMware knowledge base on CBT inconsistency after resizing a VM disk in vSphere 8.0 U2 documents that a change to how disks are extended in vSphere 8.0 Update 2, intended to make certain disk hot-grow operations more efficient, unintentionally caused incorrect change tracking, resulting in backups that did not capture the right data and, consequently, corrupt restores; the fix shipped in ESXi 8.0U2b (build 23305546). The issue only appears if a backup runs after a disk is hot-extended while the VM stays powered on; a disk resize while the VM is off does not trigger it.
A separate, earlier defect affected vSphere 6.5 and 6.7: IBM’s support documentation on VMware VM backups being corrupted when a snapshot exists while CBT is enabled describes a behavioral change in those releases that could cause VMware to present invalid CBT data to backup applications, resulting in undetected corruption of the backup copy and every subsequent incremental built on it, even though the backup reported success. Red Hat’s Customer Portal documentation of the ESXi 6.0 CBT defect separately covers a related issue where CBT returns incorrect changed sectors, which VMware resolved with a patch.
If corruption appears after a known ESXi upgrade, or after any VM in the chain had a disk resized while powered on, CBT invalidation is the first thing to check, and the fix is to reset CBT and force a fresh full backup, not to keep restoring the same broken incremental chain.
2. Snapshot consolidation was interrupted or blocked by a file lock
Every VMware-API-based backup creates a snapshot, reads from it, then consolidates (merges) the delta back into the base VMDK. If that consolidation is interrupted by a host crash, a stuck backup proxy holding a lock, or a manually cancelled task, the disk can be left in an inconsistent, partially-merged state that then gets picked up by the next backup or restore.
Broadcom’s knowledge base article on snapshot consolidation failing due to locks held by third-party backup software describes exactly this failure mode: consolidation can fail because a backup solution's proxy VM still has the disk attached, and resolving it requires identifying that proxy VM and unmounting the specific locked disk before consolidation can proceed. A related Broadcom article on VMs that fail to power on after an incompletely consolidated snapshot lists the realistic root causes, including ransomware activity that interrupts or corrupts consolidation, an unexpected ESXi host shutdown mid-consolidation, and manual termination of a consolidation task.
A "consolidation needed" warning in vCenter is not cosmetic; it is a signal that the disk chain is not in its final, trusted state. Any backup taken while that warning is active should be treated as suspect until the consolidation completes cleanly.
3. VMFS or underlying storage corruption, unrelated to backup software
Sometimes the VMDK was never intact to begin with, because the datastore it lived on suffered metadata corruption from a bad shutdown, a failing RAID controller, or faulty HBA firmware, and the backup software faithfully copied already-damaged data.
Broadcom documents a case involving HPE-rebranded Qlogic HBAs where firmware incorrectly replayed stale I/O requests, including DMA transfers to already-freed memory locations, causing heap corruption that propagated to disk and resulted in lost writes and VMFS resource-cluster metadata corruption. A separate Broadcom article on VMFS datastore corruption from overwritten data walks through diagnosing overwritten VMFS metadata, explaining that a foreign filesystem header found where VMFS metadata should be indicates an external process attempted to format or initialize the LUN, overwriting the volume's lock and heartbeat regions; the recommended diagnostic tool is the vSphere On-disk Metadata Analyzer (VOMA), used to validate and, where possible, repair the volume.
This class of corruption is not fixed by anything on the backup side — CBT reset or a new full backup does nothing if the datastore itself is damaged. Check vobd.log and vmkernel.log for corruption or heartbeat-region warnings on the source datastore before assuming the backup chain is at fault.
How to Fix It: Step-by-Step Remediation by Root Cause
Once the troubleshooting table above points to a likely cause, the remediation path differs completely by cause — resetting CBT does nothing for VMFS corruption, and running VOMA does nothing for a consolidation lock. Use the section that matches the diagnosis.
Fix path A: CBT is reporting the wrong changed blocks
1. Confirm the trigger: check whether the affected VM had a disk hot-extended while powered on, or whether the ESXi host is on a build older than 8.0U2b (build 23305546) or an unpatched 6.5/6.7/6.0 release with the documented CBT defects.
2. Schedule a maintenance window and power off the affected VM. CBT's advanced settings cannot be safely changed on a running VM.
3. Remove any existing snapshots on the VM before changing CBT settings — an active snapshot chain can cause the reset to silently not take effect. This step is the one most often skipped, and skipping it is the most common reason a "CBT reset" doesn't actually fix anything.
4. Disable CBT: in the vSphere Client, go to Edit Settings → VM Options → Advanced → Edit Configuration and set ctkEnabled to false for the VM and for each virtual disk. Via PowerCLI: Get-VM "VMName" | New-AdvancedSetting -Name ctkEnabled -Value $false -Confirm:$false.
5. Power the VM on, then off again, to clear any stale -ctk.vmdk tracking files, then re-enable CBT by setting ctkEnabled back to true the same way.
6. Patch the ESXi host to the fixed build before resuming normal operations, if the defect is version-specific.
7. Run the next backup job as a forced full (not incremental) backup — this is the step that actually replaces the invalid data, since CBT reset alone doesn't repair backups already taken.
8. Restore that new full backup to an isolated environment and validate it (see the verification workflow below) before treating the backup chain as trustworthy again.
Fix path B: Snapshot consolidation is interrupted or locked
1. Confirm the VM shows a "Consolidation needed" status in vCenter (visible under the VM's Summary tab or via Get-VM | Get-View in PowerCLI checking Runtime.ConsolidationNeeded).
2. Identify what is holding the disk lock. Check recent tasks for any backup jobs still running against this VM, and check whether a backup proxy VM has the disk attached via HotAdd — this is the single most common cause of a stuck consolidation. Skipping this identification step and repeatedly retrying consolidation without releasing the lock is the most common failure loop administrators get stuck in.
3. If a proxy VM is holding the disk, detach/unmount that specific disk from the proxy VM first, without deleting anything.
4. Retry consolidation: right-click the VM in vCenter and select Snapshots → Consolidate, or trigger it via the API.
5. If consolidation still fails, check the datastore browser or SSH into the host to look for orphaned delta/snapshot files that no longer match the VM's current snapshot list — these require careful manual reconciliation and are best done with VMware support engaged if the VM is production-critical.
6. Once vCenter confirms consolidation is complete and the warning clears, take a fresh full backup of the VM before trusting any further incrementals built on the previously locked chain.
Fix path C: VMFS or underlying storage is corrupted
1. Stop further writes to the affected datastore immediately — migrate powered-on VMs off it if possible, since continued I/O can make a marginal corruption worse.
2. SSH into an ESXi host that can see the datastore and identify the device: esxcli storage vmfs extent list to map the datastore to its underlying device.
3. Run a read-only check with VOMA (do not run a repair on the first pass): voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxxxxxx:1. Running VOMA in fix mode before reviewing the check output is the step most likely to make an already-damaged volume worse — always check first.
4. Review the output for the specific error category (heartbeat region, resource cluster, file descriptor table) — this determines whether a repair is realistic or whether the volume should be treated as unrecoverable.
5. If VOMA reports errors that are flagged as fixable, run it with the fix flag on a snapshot or clone of the LUN where possible, not directly on the only copy of production data.
6. Separately investigate the trigger: check HBA/RAID controller firmware against the VMware Hardware Compatibility List, and review vmkernel.log and vobd.log around the time corruption was first logged for storage-layer warnings.
7. Do not rely on incremental backups taken from this datastore during the suspected corruption window. Restore from the most recent full backup confirmed to predate the corruption, then validate it before returning the VM to production.
Verify a Restored VMDK Before You Trust It
Before promoting a restored VM back into production, confirm it is actually intact rather than assuming the restore succeeded because it completed without an error message.
A horizontal workflow diagram showing five stages — Restore to isolated host/network → Boot and check guest OS + event log → Run file-system/application-level checks → Compare checksums or record counts against a known-good reference → Promote to production or quarantine for further diagnosis — with a branch at the checksum stage leading back to "diagnose backup chain" if validation fails.
How to Stop It Happening Again: Prevention Checklist
Reset CBT after any "disk re-open" event. Hot-extend, storage vMotion, and certain host-crash recoveries are documented triggers for CBT going stale. Broadcom's documented reset procedure is to power off the VM, confirm there are no active snapshots, disable CBT for the VM and each attached virtual disk via the advanced configuration parameters, then re-enable it and force a full backup afterward rather than trusting the next incremental.
Keep ESXi hosts patched to the build that fixes known CBT defects — for the 8.0 U2 hot-extend issue specifically, build 23305546 (8.0U2b) or later.
Don't schedule a backup immediately after a live disk resize. If a hot-extend is unavoidable, force a disk re-open (power cycle, snapshot-and-remove, or suspend/resume) before the next backup runs.
Treat "consolidation needed" as a blocking condition, not a background warning. Resolve locks — including checking backup proxy VMs for HotAdd-mounted disks — before the next backup job runs against that VM.
Schedule periodic full (not only incremental-forever) backups. A full backup reads every block directly and does not depend on CBT's change list being correct, which caps how much damage a silent CBT defect can do.
Run VOMA or equivalent datastore health checks periodically, especially after unexpected host shutdowns, RAID rebuilds, or HBA firmware updates, rather than only after corruption is already suspected.
Schedule actual test restores, not just backup-success monitoring. NIST's Cybersecurity Framework 2.0 recovery guidance is explicit that the integrity of backups and other restoration assets should be verified before they are used for restoration, and NIST SP 800-53's contingency-planning control CP-9(2) recommends periodically restoring a sample of backup data specifically to confirm it is reliable, not just present.
Should You Trust This Backup, or Verify First?
Condition present | Risk level | Recommended action |
ESXi host recently upgraded to 8.0 U2 (pre-U2b), and any VM disk was host-extended while powered on | High | Reset CBT, force a full backup, verify with a test restore before relying on the resulting chain |
Host running unpatched vSphere 6.5/6.7 with a documented CBT behavioral defect | High | Patch to the fixed release, reset CBT, force a full backup |
vCenter shows "consolidation needed" on the source VM | High | Resolve the lock and complete consolidation before trusting any backup taken while the warning was active |
Recent unexpected host shutdown or RAID/controller event on the source datastore | Medium-High | Run VOMA in read-only check mode on the datastore before assuming the backup chain itself is at fault |
Guest boots and file system is clean, but a specific database/application reports inconsistency | Medium | Check quiescing/VSS writer health rather than the backup chain; consider re-enabling application-consistent snapshots |
None of the above; routine incremental chain on a patched, stable host | Low | Standard restore is reasonable, but a periodic sampled test restore per NIST SP 800-53 CP-9(2) is still good practice |
Because these failure modes are silent by design, the practical mitigation is procedural: schedule periodic active full backups instead of running incremental-forever indefinitely, and validate restores rather than only monitoring job status. Vinchin Backup & Recovery supports configurable full-backup scheduling alongside incremental and differential strategies for VMware environments, so teams can enforce a periodic full-backup cadence without relying solely on CBT-derived incrementals for VMs that have gone through a resize, host migration, or upgrade event.
FAQs
Q1: Can vSphere Replication corrupt a VM the same way a backup restore can?
Replication uses a different change-tracking path than backup software calling the CBT API, so it isn't exposed to the same CBT-invalidation bugs described here. It can still propagate application-level corruption from the source VM, though, and it doesn't protect against ransomware or accidental deletion the way a separate backup copy does.
Q2: Does thin vs. thick provisioning change the risk of restored-VMDK corruption?
Provisioning type doesn't change whether CBT returns correct data, since CBT operates above the provisioning format. It can affect how a hot-extend is carried out, but the documented CBT-after-resize defect is about the resize event itself, not the disk format.
Q3: Is a corrupted restored VMDK the same problem as a corrupted snapshot?
No. A corrupted snapshot is a live-environment problem, a delta file or CID chain breaks on the running datastore. A corrupted restored VMDK means the backup copy in a separate repository was already invalid before the restore began; the restore just surfaces it.
Q4: Could ransomware encryption look like VMDK corruption after a restore?
Yes, and it's worth ruling out first. An encrypted guest filesystem restored from a backup taken after infection will boot to unreadable data that looks like structural corruption. Check backup timestamps against known indicators of compromise, and test-restore from a point clearly before the suspected infection window.
Q5: Does patching only vCenter Server fix the CBT issues described here?
No. These are ESXi host-level defects. vCenter orchestrates and displays the snapshot and backup tasks, but the change-tracking data itself is generated by the ESXi kernel, so the fix has to be applied to the affected hosts, not just vCenter.
Conclusion
A restored VMDK is corrupted almost every time because the backup was already invalid before the restore ran, through silent CBT invalidation, an interrupted snapshot consolidation, or storage-level damage the backup software only copied. Fixing the immediate restore matters less than identifying which of these produced it. Pairing periodic full backups with scheduled test restores catches the failure before a real recovery depends on it.
