Series: Building Resilient SAP Infrastructure on VCF
In Part 1, I laid out the four-site, four-layer architecture. In Part 2, I built the floor: two stretched domains across Site A and Site B, vSAN ESA under the management plane, NetApp MetroCluster NFS under the SAP workloads, and three referees at Site C. Everything in those two posts has one job: keeping virtual machines and datastores alive through a site failure. None of it knows what a HANA primary is.
This is the SAP HA on the VCF layer, where the design starts making decisions about roles. VMware HA can restart a VM on a surviving host. It cannot decide which of two HANA systems should be accepting writes; it cannot tell an ASCS instance that its enqueue lock table now lives somewhere else, and it will happily restart a database VM at the wrong site while its replication partner is already serving the business. This is the layer most people mean when they say “the HA design”, and it is the one I promised had the sharpest edges.
This series is based on a design that has not yet gone through build and test. What follows is the set of decisions made on paper, what changed when I checked them against the vendor requirements, and what remains open going into the build phase. Numbers that depend on testing are marked as such.
Why VMware HA Is Not Enough for HANA
Take a 500 GB HANA primary at Site A and lose the host under it. VMware HA restarts the VM on another host in a minute or two. That is the easy part. Then HANA starts, and that is where the clock really begins. A cold start has to load column tables from disk before the database can serve the workload at full speed, and on a database of this size that takes tens of minutes, not seconds, even when the storage delivers several GB/s.
A preloaded secondary at Site B, kept in sync by HANA System Replication, already has the tables in memory. Takeover means promoting it and redirecting the clients. That is the whole argument for running a cluster on top of a cluster, and it is why Part 1 quoted takeover in minutes rather than the fifteen to thirty minutes a reload from disk can take.
So the design stacks four decision layers, each answering a different question:
The lower two were Part 2. The upper two are in this post. They run on SLES for SAP Applications, which brings the High Availability Extension, Pacemaker, Corosync, SBD, and the SAP resource agents in one package. Standard SLES does not, and that distinction matters when someone asks why the “same” operating system costs more.
If your landscape runs RHEL for SAP Solutions instead, the HANA resource agents differ: Red Hat ships SAPHana and SAPHanaTopology in the resource-agents-sap-hana package rather than SUSE’s SAPHanaSR-angi, and the parameter names are close but not identical. Everything else in this post, fence_vmware_rest, SBD, the QDevice vote, and the whole fencing argument, is the same on both distributions. I verified the details below against SLES for SAP because this design uses it.
HANA System Replication in SYNC Mode
Each production HANA system runs as a pair: primary at Site A, secondary at Site B, both on the same stretched workload cluster, each with its own dedicated NFS data and log datastores. HANA handles replication over the Private VLAN on the metro wave, with roughly 0.5 ms round trip.
HSR offers three replication modes and one option that matters here.
SYNC waits until the secondary has written the redo log to its own log volume before the primary acknowledges the commit. SYNCMEM only waits for the log to reach the secondary’s memory. ASYNC does not wait at all. For RPO 0 across two sites, SYNC is the only choice, and at 0.5 ms the extra write round trip disappears into the same noise as the MetroCluster write path I walked through in Part 2. SYNCMEM buys a fraction of a millisecond and gives back the guarantee that a power loss at the secondary site cannot lose committed transactions. Not worth it.
The option is FULLSYNC. With it on, if the secondary becomes unreachable, the primary stops committing until it comes back. That turns a secondary problem into a production outage, which is the opposite of what a stretched design is for. It stays off. The cost of leaving it off is a window during a secondary outage when the primary runs without replication protection. Pacemaker tracks that state, which is exactly why the cluster refuses to fail over to a secondary that wasn’t in sync at the moment of failure.
The secondary preloads column tables into memory so that takeover is a promotion rather than a reload. That is why every HANA primary in the Part 2 sizing has a secondary with the same memory footprint, and why the 3,200 GB workload number is what it is.
One caveat sits on top of this table. Under heavy interfacing load, takeover can run longer than the figures SAP publishes for the planned-maintenance case, and no design document can answer that in advance. Takeover time is measured under representative load during build testing, and nothing in this post should be read as a committed RTO for the HANA layer.
Pacemaker: Two Nodes and a Referee at Site C
Each HANA pair runs a two-node Pacemaker cluster, one node per site. On SLES for SAP, the package is SAPHanaSR-angi. SUSE states that new deployments should use angi rather than the classic SAPHanaSR package, and SLES for SAP 16 removes the classic package, so a build starting in 2026 starts on angi. It brings three resource agents: SAPHanaTopology, which watches the replication state on both nodes; the promotable SAPHanaController, which decides which node holds the primary role; and SAPHanaFilesystem, which monitors /hana/shared and is worth having here because /hana/shared is on NFS.
Two parameters drive the cluster’s behavior. PREFER_SITE_TAKEOVER=true tells the cluster to promote the in-sync secondary at the other site rather than restart the failed primary in place, which is the point of having a secondary. AUTOMATED_REGISTER decides whether the old primary, once it is back, automatically registers itself as the new secondary. Leaving it false means a human confirms the former primary is safe before it rejoins replication, at the cost of running unprotected until someone does. That is a policy question for operations, not infrastructure, and it stays open until the build.
A two-node cluster cannot vote. When the two nodes lose sight of each other, each holds one vote out of two, and neither can claim quorum. The third vote is Corosync QDevice, with the qnetd daemon at Site C. In the current design, it is planned on the same small Linux VM (2 vCPU, 4 GB RAM) as the ONTAP Mediator, because both are tiny and both need exactly the same thing: an independent failure domain with its own network path to each primary site. Whether NetApp supports sharing the Mediator host with another service is still being confirmed. If not, qnetd gets its own VM of the same size, and nothing else changes.
Corosync, the Pacemaker heartbeat, and HANA replication all run on the same Private VLAN across the metro wave. Corosync totem timeouts for a stretched cluster are deliberately not set in the design. The defaults are tuned for nodes in one rack, and a metro link with maintenance windows on the DWDM equipment is a different environment. The values are a build-phase decision, after the link has been characterized, not copied from a guide written for a different topology.
Placement rules at the VMware layer back all of this up: HANA primary and secondary are never on the same ESXi host, ASCS and ERS are never on the same host, and ASCS and the primary application server are never on the same host. Those are hard anti-affinity rules. Site preference stays as “should run” rules, exactly as in Part 2, so vSphere HA can still cross sites when a site failure leaves it no choice.
fence_vmware_rest: The Part With the Sharpest Edges
Fencing is the mechanism that stops a node that might still be writing before another node takes its role. Without it, a cluster that believes a node is dead can promote a secondary while the old primary is still alive and still accepting commits, and now you have two primaries and a very long weekend. Pacemaker will not promote without a successful fence. That makes the fencing path the single most important path in the SAP layer, and it is the one with the most ways to fail quietly.
Why hardware fencing was rejected on day one
The classic approach is a power fence through the server’s out-of-band controller: iLO, iDRAC, IPMI. In a physical SAP landscape, that is correct. Here, every cluster node is a virtual machine. An IPMI fence powers off the ESXi host, and with it every other SAP VM that happens to be on that host. Fencing one HANA node by taking down an application server, a Web Dispatcher, and possibly the other cluster’s ERS is not fencing; it is a self-inflicted outage. Fencing has to work at the VM level.
What fence_vmware_rest does
fence_vmware_rest is the fence agent in the standard fence-agents package that powers a VM off through the vCenter REST API. It only works against vCenter, not directly against an ESXi host, and the configuration maps each Pacemaker node name to its VM name in vCenter through pcmk_host_map. When Pacemaker decides a node must go, the agent calls vCenter, vCenter powers off exactly that VM, and nothing else on the host is touched.
It needs a vCenter service account. The account should be scoped to the SAP cluster VMs and to power operations on them, nothing wider. That account sits on the infrastructure side of the boundary rather than the SAP side, which is worth noticing early. Small detail, real dependency: without that account, the cluster cannot fence, and without fencing, it will not fail over.
The dependency that makes it sharp
fence_vmware_rest can only fence if vCenter answers, and only if vCenter can reach the ESXi host that runs the target VM. Both conditions fail in a site failure, in two different ways.
First, vCenter is itself a VM. The workload domain vCenter runs in the stretched management domain from Part 2, on vSAN, and it can be at Site A when Site A fails. Play that out: the HANA primary at Site A is gone. Pacemaker at Site B, still holding quorum with the QDevice, wants to promote the secondary. To do that, it must fence the Site A node first. It calls vCenter. vCenter is also gone, and it stays gone until vSphere HA restarts it at Site B, which takes minutes. During that window, the cluster is stuck: it knows what to do and cannot prove it is safe to do it.
Second, and this is the part that turns SBD from a backup into the mechanism that matters: even with vCenter alive and well at Site B, it cannot power off a VM on an ESXi host it can no longer reach. The Site A host is disconnected, the VM’s power state is unknown, and the power-off call fails. It keeps failing until vSphere HA restarts that VM somewhere at Site B, at which point the fence becomes a power-off of a VM that HA just brought back. In every real site failure, fence_vmware_rest doesn’t complete the takeover. It handles the everyday cases well: a hung node, a kernel panic, a VM that lost its cluster network but not its host. For the site failure the design exists for, something that does not need the management plane has to finish the job.
This is not a bug in the fence agent. It is what “fencing through the management plane” means in a stretched cluster, and it is why the design includes a second fencing mechanism, not as decoration.
SBD as the second mechanism
SBD, storage-based death, uses a shared block device as a message channel. A node that must be fenced gets a poison message written to its slot on the SBD disk, and a watchdog on that node guarantees it resets itself if it does not confirm it is still alive within the timeout. SBD does not need vCenter. It needs the shared disk to be reachable and a watchdog inside the VM, which on a virtual machine means the softdog kernel module, since there is no hardware watchdog to pass through.
With disk-based SBD, Pacemaker supports fencing levels, so the configuration is fence_vmware_rest first, SBD if that fails. In the vCenter-is-down scenario above, the REST call times out, the cluster falls through to SBD, and takeover proceeds.
Here is the open item. Disk-based SBD needs a block device, and the principal storage for a VCF 9 greenfield workload domain is NFSv3 only. The SBD disk cannot sit as a file on the NFS datastores. One option is a small iSCSI LUN served from the same AFF controllers, mirrored by MetroCluster so it survives a site failure, presented directly to the cluster VMs in the guest. It touches the NetApp configuration, the in-guest iSCSI setup, the MetroCluster switchover behavior for that LUN, and the SBD msgwait and watchdog timeouts, which must be longer than any MetroCluster switchover the LUN might go through; otherwise, SBD resets a healthy node during a storage event.
The alternative being evaluated is diskless SBD: softdog watchdog plus stonith-watchdog-timeout, with no shared disk at all. SUSE’s High Availability Administration Guide is explicit that a two-node cluster may only use diskless SBD when QDevice is also configured, which this design has. The node that loses quorum resets itself, and Pacemaker treats it as fenced once the watchdog timeout has passed. That removes the whole iSCSI LUN and switchover-timing problem from the list.
One correction to the picture of “levels” is needed here. Diskless SBD is not a second fencing level in the topology sense. No stonith resource falls through after fence_vmware_rest fails. The cluster relies on a timeout when no fence device can confirm the kill: after stonith-watchdog-timeout, Pacemaker assumes the unreachable node has reset itself, whether or not the REST fence was attempted. The trade-off is that there is no remote fence message; the mechanism is quorum-loss self-fencing only, and the takeover waits for the full watchdog timeout before it can proceed. SUSE also requires SBD_WATCHDOG_TIMEOUT to be longer than the QDevice sync_timeout, or SBD will not start, which ties the two timers together and adds another number to set with the metro link in mind.
Neither option is finalized. Both are on the list, and the choice will be based on measured takeover time, not on paper.
The vote can go to the wrong site
With QDevice in place, there is no fence race in the classic two-node sense. Only the node that receives the third vote has quorum, and no-quorum-policy stops the other side from fencing anyone. The sharp edge is the opposite one: which side gets the vote.
In a clean, symmetric partition, where both nodes can still reach qnetd but not each other, the QDevice picks a side by tie-breaker, and the default tie-breaker is the lowest node ID. If the HANA primary happens to be node 2, a link failure with both sites healthy promotes the secondary for no reason at all, with a takeover the business did not need. The fix is to configure the tie-breaker, or QDevice heuristics, so the vote goes to the site running the primary, with priority-fencing-delay as a backstop so that the node carrying the promoted resource never loses a close call. The values are a build-phase decision. The rule is fixed: the losing site fences itself, never the winner, and the winner is the site with the primary unless it is actually dead.
The Partition Nobody Wants: Three Referees, One Answer
Site failure is the easy case. Everything at one site is dead; every referee agrees; the survivor wins. A network partition between Site A and Site B, with both sites alive, is the ugly one, because three independent arbitration mechanisms make their decision at the same time:
They do not talk to each other. Nothing in Pacemaker knows what MetroCluster is doing, and nothing in vSphere HA knows what the QDevice decided.
Storage behavior in this scenario is what people get wrong. On an ISL loss with both sites alive, the Mediator does not switch anything over. Each site’s controllers keep serving the aggregates they own, mirroring stops, and the datastores stay online at the site that owns them. The risk is locality, not switchover: an ESXi host at Site B can no longer reach a datastore whose SVM lives at Site A, because the stretched storage network it used to get there is the link that just failed. If the QDevice gives quorum to Site B and Pacemaker promotes the HANA secondary there, that secondary had better be on datastores owned by Site B, or it is being promoted onto storage it cannot see.
So the design rule has two halves. The first is one Part 2 did not state, because Part 2 showed the datastore layout from the primary side only.
Design rule: the secondary owns its own storage. With HSR, every secondary has its own data and log volumes, separate from the primary’s. Put those volumes on an SVM owned by the secondary’s site controllers, mirrored back to the primary site. A promotion at Site B then runs entirely on Site B storage when the metro link is cut, and never depends on a datastore owned by the other side of the failure.
The second half is that all three referees must see the same thing from Site C: independent, symmetrical paths from Site C to each primary site, so that whichever site loses its path to Site C is the site that loses at every layer.
The Gap That Was Found on Paper
That last requirement is where a first pass at Site C turned out to be wrong.
The obvious way to connect a third site is a single link to wherever the main network already terminates, and in this case that meant one 1 Gbps circuit landing at Site A. Bandwidth was never the question. The Mediator needs tens of megabits, and the QDevice needs almost nothing. The topology was the question. If Site A dies, Site B loses its replication peer and its access to both referees instantly, because the only path to Site C ran through the site that just failed. The Mediator and the QDevice would be unreachable at exactly the moment they exist to answer.
On paper, that design satisfied every latency and bandwidth requirement from Part 2. It failed the independence requirement, which matters, and which NetApp states explicitly for the Mediator. So the requirement changed: Site C needs a path to Site B that does not traverse Site A. Bandwidth is trivial; routing is not, and it’s worth insisting on before anything is cabled, because it is far cheaper to specify a second path now than to discover the dependency during a real site failure.
The second point follows from the first. A Site C failure on its own does not stop production. HANA keeps replicating, MetroCluster keeps mirroring, the SAP VMs keep running. What it does is silently remove split-brain protection from all three layers at once. That state needs to page someone. A yellow icon on a dashboard is not enough for “the estate is currently running without a tiebreaker”, so loss of the Mediator or the QDevice is an urgent alert in this design, not an informational one.
ENSA2: The Application Layer Failover
Above the database sits the SAP central services instance, ASCS, which owns the enqueue server and the lock table every application server depends on. Lose the lock table and every open transaction in the system is gone. ENSA2, the Enqueue Replication Server 2, replicates that table to an ERS instance on another node continuously, so that when ASCS fails, Pacemaker can start it on a surviving node and the enqueue server rebuilds its table from the replica.
ENSA2 removed the old ENSA1 rule that ASCS had to fail over to exactly the node where ERS was running. With ENSA2, ASCS can start on any cluster node and pull the table from the ERS over the network, simplifying placement constraints and making failover more predictable. The failover target in this design is 30 to 60 seconds with the lock table preserved. The anti-affinity rules from earlier keep ASCS and ERS on different ESXi hosts, because a replica on the same host as the original protects against nothing.
The SAP instance directories and the sapmnt share live on the NetApp NFS, mounted at both sites, so the surviving node has the binaries and profiles without any cluster-managed filesystem failover.
What makes the HANA takeover close to invisible from the application side is on the NetWeaver layer, not the cluster: connectivity suspend (SAP Note 1913302). The application servers hold their database connections and retry, rather than dropping user sessions, while the takeover completes. Users see a pause. They do not see a login screen as long as the takeover finishes within the configured suspend timeout, which is another reason the measured takeover time matters.
Small Things That Block Big Things
DNS. Every Pacemaker virtual IP, every HANA virtual hostname, and every cluster node name must resolve from both sites before go-live and before the cross-vCenter vMotion that brings the landscape onto this platform. It is a five-minute task that stops the migration cold if it is missing, because the SAP resource agents check hostnames on start and refuse to run against names they cannot resolve.
The vCenter service account for fencing, already mentioned, is in the same category. So are the SBD decision and the Corosync timeouts. None of these is difficult. All are on the critical path, and each sits in a different part of the stack.
SAP HA on VCF: Where This Leaves Us
The SAP HA on VCF layer in this design is mostly conventional: HSR in SYNC mode, SAPHanaSR-angi, a QDevice for the third vote, ENSA2 above it. If you have built a HANA cluster on physical servers, most of it will look familiar. The decisions I would defend hardest are the ones about fencing, because that is where a virtualized SAP landscape differs from a physical one, and where copying a physical reference design would produce something that looks right and fails on the day.
Rejecting IPMI fencing is the obvious one. Putting SBD behind fence_vmware_rest is the one I think most designs skip, and the reason it cannot be skipped is that in a real site failure the REST fence never completes: vCenter may be at the dead site, and even when it is not, it cannot power off a VM on a host it cannot reach. And treating Site C’s network independence as a hard requirement, rather than a bandwidth line item, is what turned out to matter before a single host was racked, because on paper the third site was fine and in practice it would have died with Site A.
This post in the series has the most open items. Five are open going into the build:
Those five get a post of their own once the build tests have run: what the fencing tests showed, the measured takeover time, the SBD decision, and the tie-breaker configuration that survived a real partition. That post sits between this one and Part 4, because Part 4’s lessons learned depend on it.
What’s Next
In Part 4: DR, Ransomware, and Lessons Learned, the design leaves the two primary sites and goes to Site D, with Veeam and Object First Ootbi as the immutable target. The scenario is the one Part 1 set up: MetroCluster has faithfully mirrored encrypted data to both sites, HSR has faithfully replicated it into the secondary, every HA layer in this post did its job perfectly, and none of them helped. Getting back to clean from there is what Site D is for.
As always, if you have questions or want to argue with any of these choices, leave a comment or reach out to me on social media. Some of the best design improvements I have made came from someone telling me why I was wrong.
Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.
