vSphere HA has a simple job description: when an ESXi host dies, restart its virtual machines on a host that is still alive. When a host genuinely stops — power loss, a purple diagnostic screen, a dead motherboard — HA does exactly that, usually within a couple of minutes, and nobody writes an article about it.

The failures worth understanding are the ones where the host has not cleanly died, and HA has to decide what happened from incomplete information. Get that decision wrong and you get two copies of the same VM writing to the same disk, or VMs that are killed and never restarted, or a cluster that refuses to power anything on at all.

How HA decides a host is dead

One host in the cluster is elected primary (the rest are subordinates); each runs the FDM agent. Per Broadcom’s documentation, the primary monitors the others “through the exchange of network heartbeats every second.”

When those heartbeats stop, HA does not immediately declare the host dead. It runs two more checks:

  • Datastore heartbeating. Each host also writes a heartbeat to shared datastores (two, by default). If the silent host is still touching those files, it is alive — just unreachable over the management network.
  • ICMP pings to the host’s management IP.

The documentation is explicit about the two outcomes. If the host “fails to respond to pings and isn’t issuing heartbeats” to a datastore, then “the host’s virtual machines are restarted on alternate hosts.” But if datastore heartbeating continues, “the primary host assumes that the secondary host is in a network partition or is network isolated” — and keeps monitoring instead of restarting anything.

That branch is the whole ballgame. HA distinguishes three cases: failure (host stopped), isolation (host lost the network but is still running), and partition (a group of hosts can still talk to each other but not to the primary).

Isolation, and the split-brain risk

A host declares itself isolated when it stops seeing heartbeats from other hosts and cannot reach its isolation address (by default, the management gateway). The isolated host still has running VMs. What it does with them is the host isolation response, and it is a genuine dilemma:

  • Leave powered on — assume this is a transient network blip and don’t touch the VMs.
  • Power off and restart — hard-stop the VMs (equivalent to pulling their virtual power) so they can be restarted elsewhere.
  • Shut down and restart — a graceful guest shutdown via VMware Tools, falling back to power-off after a timeout (das.isolationshutdowntimeout, 300 seconds by default).

“Leave powered on” exists to protect against the false positive: the host thinks it is isolated when the real problem is elsewhere, and killing healthy VMs would make an outage out of a non-event. But it creates a different hazard. If the host is truly isolated — including from storage — the surviving hosts will try to restart those VMs, and can only succeed once the isolated host’s lock on the VMDK files times out. Until then you have a window where the VM exists in two places. vSphere resolves this by having the isolated host kill the original (“ghosted”) VM once it detects it can no longer reacquire the disk locks — but the safety net is the storage lock, not HA’s own logic.

Partition

A partition is isolation’s larger cousin: the cluster splits into groups that can each talk internally but not across the divide. Each partition with no primary elects its own. Now two primaries are independently deciding which VMs need restarting, arbitrated only by datastore access and file locks. If the partition also cuts storage, the same ghost-VM cleanup applies. If it doesn’t, HA generally rides it out — but you are running a cluster whose control plane briefly disagreed with itself, and vCenter’s view of “where is this VM” can lag reality until the partition heals.

Storage that half-disappears: APD vs PDL

The nastiest ambiguity is storage. VM Component Protection (VMCP) splits it into two cases, and the distinction matters:

  • PDL — Permanent Device Loss. The array actively tells ESXi, via SCSI sense codes, that a device is gone for good (a LUN was unmapped, a volume deleted). This is unambiguous, so the response can be immediate: power off the affected VMs and restart them on a host that still has the datastore.
  • APD — All Paths Down. Storage simply stopped responding, with no word on whether it is coming back. ESXi cannot tell a 30-second SAN reconverge from a dead fabric. It waits out a timer (Misc.APDTimeout, 140 seconds by default), then a further “delay for VM failover for APD” (3 minutes by default) before acting.

The APD response is itself a conservative-vs-aggressive choice: conservative only terminates a VM if HA believes it can be restarted elsewhere; aggressive terminates it regardless. Aggressive on a cluster-wide storage outage — where no host can restart the VM — means you have killed your workloads and gained nothing. Conservative on a single-host storage fault is exactly right. There is no setting that is correct for both.

When HA refuses to power anything on

The last failure mode isn’t a crash — it’s HA doing its job too literally. Admission control reserves enough spare capacity to restart VMs after a configured number of host failures, and it enforces that by blocking power-on operations that would eat into the reserve.

With the default slot policy, HA sizes a “slot” to the largest CPU and memory reservation among running VMs. Set one big memory reservation on one VM — or deploy an OVA that ships with reservations baked in — and the slot size jumps, the slot count drops, and suddenly you cannot power on VMs that fit the cluster comfortably. The error — “Insufficient failover resources” — points at HA, but the cause is one reservation and a policy that extrapolates from the worst VM. The percentage-based policy avoids the slot-size cliff and is the usual recommendation for mixed clusters.

The lessons that generalise

HA restarts VMs; it does not keep them running. Every HA event is a hard reboot of the guest. Databases come back needing crash recovery, non-clustered apps drop every in-flight request. HA bounds your downtime to minutes — it does not make the failure invisible.

Ambiguity is the hard case, and it’s unavoidable. Isolation response, conservative vs aggressive APD, leave-powered-on vs power-off — each is a bet on false-positive versus false-negative. The right bet depends on your network and storage redundancy, which means the defaults are a guess about your environment.

Storage isolation is worse than network isolation. A host that loses only the management network keeps serving its VMs. A host that loses storage — or a cluster that does — pushes HA into its most destructive responses, and the protection against split-brain is a disk lock timing out, not HA reasoning about it.

Admission control is a promise you have to fund. “Tolerate one host failure” means one host of capacity sits idle. If that math stops working — utilisation crept up, someone added a large reservation — HA keeps the promise by refusing new workloads, and it will not ask first.

Tech-Bee is an independent technology publication. This piece is drawn from Broadcom’s vSphere Availability documentation and knowledge-base articles, plus Duncan Epping’s Yellow Bricks write-ups on HA internals. Default timer values are version-dependent — check your own cluster’s advanced settings.