Series: Building Resilient SAP Infrastructure on VCF
In Part 1, I laid out the overall architecture: four sites, four layers of high availability, and a separate recovery site built on Veeam and Object First Ootbi. That post was the map. This blog post Building Resilient SAP Infrastructure on VCF — Part 2: Compute and Storage is where we start walking the ground floor, because the compute and storage layer is where every good HA design is either made or quietly undermined.
I want to be honest about how I approach a post like this. I am not going to give you a click-by-click build guide. What I find useful, and what I wish more design content did, is explain why each decision was made, what the alternatives were, and where the traps are hiding. A design is only as good as the reasoning behind it, and reasoning is the part you cannot copy from a reference architecture.
Everything here was validated against VCF 9.0 and still holds on VCF 9.1.
Two Domains, and Why I Split the Storage
VCF gives you two kinds of domains. The management domain runs the control plane: vCenter, SDDC Manager, NSX Manager, and VCF Operations. The workload domain runs your actual workloads, in this case, the SAP landscape, with its own vCenter, its own clusters, and its own storage.
That much is standard. The decision I want to explain is not automatic: I run these two domains on two completely different storage technologies. The management domain sits on vSAN ESA. The SAP workload domain sits on NetApp MetroCluster NFS. Both stretched across the two primary sites, but on entirely separate stacks.
The obvious question is why not keep it simple and use one storage platform for everything. Either vSAN everywhere or NetApp everywhere. It would have fewer moving parts and one less thing to license and learn.
I split them on purpose, and the reason comes down to blast radius. If the management plane and the SAP workloads share the same storage, a storage problem doesn’t just take down your databases; it takes down the tools you use to fix them. That is the worst possible moment to lose vCenter and NSX Manager. By putting the management domain on vSAN ESA, which lives on the local NVMe inside the ESXi hosts, the management plane has no dependency on the NetApp array at all. NetApp can be in a maintenance window or having a genuinely bad day, and I still have full control of the environment.
Meanwhile, the SAP workload domain gets a storage platform built for this: MetroCluster with synchronous mirroring, automatic switchover, dedicated all-flash performance, and the ONTAP ransomware protection I covered in Part 1. Those are things vSAN does not give me in the same way, and they matter far more for a HANA database than they do for vCenter.
So the split is not complexity for its own sake. It is two different problems getting two different tools, each one chosen for what that domain actually needs.
There is also a hard constraint worth knowing because it removes a choice you might think you have. For a greenfield VCF 9.x deployment, if you want to stretch the management domain, vSAN is the way to go. Broadcom’s vMSC guidance (KB 417356) states plainly that a vSphere Metro Storage Cluster is not supported for the initial cluster of the management domain. In other words, you cannot stretch the management domain on external NFS storage in a clean greenfield build. So even if I had wanted to put management on NetApp, the supported path would have pushed me to vSAN stretched for that domain anyway. The design and the product rules happen to agree here, which is always a comfortable place to be.
| Domain | Storage | Stretched | Quorum / witness |
|---|---|---|---|
| Management domain | vSAN ESA (local NVMe) | Yes, Site A + Site B | vSAN witness at Site C |
| SAP workload domain | MetroCluster NFS (external) | Yes, Site A + Site B | MetroCluster Mediator at Site C |
How the Stretched Clusters Work, and the Part VCF Does Not Do For You
Both domains stretch across the two sites, but they get there differently. The management domain uses vSAN to synchronously mirror data between the local NVMe storage at Site A and Site B, with the witness at Site C breaking ties. The workload domain has all its hosts in a single vSphere cluster, sharing the same MetroCluster NFS datastores from both sites, with the Mediator at Site C authorizing storage switchover.
Here is the part that caught my attention, and the part I think a lot of people get wrong because they assume VCF is doing more than it is: VCF is not aware of vMSC. Broadcom says this outright in KB 417356. As far as VCF is concerned, a stretched workload domain cluster is just a vSphere cluster. SDDC Manager will happily provision it and manage its lifecycle, but it does not configure the metro storage cluster behavior for you. That is manual work, and if you skip it, you end up with a cluster that looks stretched and does not behave as if it were stretched when a site goes down.
The manual work that matters:
- vSphere HA admission control must reserve sufficient capacity to handle the loss of an entire site, not just a single host. This is the setting people get wrong most often.
- Host groups and VM groups, so the HANA primary lives at one site and the secondary at the other, but as “should run” rules, never “must run”.
- APD and PDL responses are configured according to your storage vendor’s guidance.
- Datastore heartbeats are chosen deliberately across both sites.
I want to underline the “should run” versus “must run” point because it is a genuine footgun. It is tempting to pin your HANA primary hard to Site A with a “must run” rule. It feels tidy. But the whole reason you built a stretched cluster is so that HA can restart that VM anywhere when a site dies. A “must run” rule takes that freedom away at exactly the moment you need it. Soft rules keep your placement clean during normal operation and let HA do its job in the event of a failure. Use soft rules.
vMotion across the metro link at sub-1ms is effectively free, so planned maintenance is genuinely non-disruptive: evacuate a site, patch it, migrate back. That alone justifies much of the design effort, because planned maintenance is far more frequent than disasters.
NSX: Required, but Not the Star Here
NSX comes with VCF, whether or not you plan to use overlay networking. It is deployed as a three-node NSX Manager cluster across the two primary sites, providing the network control plane with its own redundancy within the stretched design.
For the SAP workloads themselves, I keep them on VLAN-backed networking in most cases, and I do not consider that a compromise. HANA, application servers, and central services are more comfortable on traditional VLAN-backed port groups, where IP planning and troubleshooting are straightforward. NSX still earns its place for policy consistency and micro-segmentation, and I am glad it is there, but not every SAP network needs to run on an overlay just because the platform can. Use the tool where it helps, not everywhere it is available.
Compute Sizing: The N+1 Trap That Catches Good Engineers
This is the section I most wanted to write, because it is where careful-looking designs fail in production, and they fail quietly until the day they do not.
The rule is simple to state and easy to get wrong: a single surviving site must be able to run the entire production workload on its own. Not the two sites combined. One site, by itself, after it has already lost a host of its own.
Start with the workload. For a medium SAP landscape, it climbs fast. HANA primaries for S/4 and BW often want 500 GB or more each, and every primary has a synchronous replication secondary at the other site with the same footprint, so your HANA memory requirement doubles before you have added a single application server. Add the app tiers, central services, Web Dispatcher, and the workload domain infrastructure, and for this design, I am working with roughly 3,200 GB of RAM total. And I will say this plainly because it matters: you do not overcommit RAM on HANA. Ever. It is an in-memory database. Overcommit is not a clever optimization here; it is a production incident waiting for a quiet afternoon.
Now the calculation that actually protects you:
| Step | What you calculate | Result |
|---|---|---|
| 1. Raw capacity per site | 4 hosts x 1,536 GB RAM | 6,144 GB |
| 2. After VMware overhead (~5%) | 4 hosts x ~1,459 GB usable | 5,836 GB |
| 3. N+1 per site (lose 1 host) | 3 hosts x 1,459 GB | 4,377 GB usable |
| 4. 85% planning ceiling | 4,377 GB x 0.85 | 3,720 GB safe target |
| 5. Actual workload | All SAP VMs | 3,200 GB |
The workload falls within the 85% ceiling, with room to spare. That is where I want to be on day one.
Here is the trap, and I have seen versions of it more than once. Someone looks at two sites, each with 6,144 GB, sees more than 12 TB of capacity for a 3,200 GB workload, and concludes there is enormous headroom. On paper, there is. But that number is a fiction the moment a site fails, because then one site carries everything, and that site has also lost one of its own hosts to the N+1 reserve. The 12 TB you thought you had is really 4,377 GB of usable, survivable capacity per site. Size against that number. If you size against the raw cluster total, you have built a cluster that works perfectly right up until the failure it was supposed to survive.
Why 85% and not 100%? Because the last 15% is the margin that keeps you out of trouble. Per-VM memory overhead grows with large VMs. HANA databases grow every quarter, whether you planned for it or not. Backups add temporary overhead. And something always gets added after go-live that was not in the original sizing. If you are sitting at 95% of N+1 usable on the day you go live, you are already behind, and hardware lead times mean the fix is months away, not days.
This is also where the two-domain split quietly pays for itself in hardware. Because the management VMs reside in their own vSAN domain, they do not consume capacity in the SAP cluster. If vCenter, SDDC Manager, NSX Manager, and VCF Operations were sharing this cluster, I would likely need a fifth host per site to hold the same margin. The split is not just cleaner architecturally; it is cheaper where it counts.
One last sizing note, and this one comes straight from Broadcom’s vMSC guidance rather than from me. Host count per site affects your upgrades, not just your failures. With only three hosts per site, patching a host forces one of your three NSX Manager nodes to migrate across to the other site during the maintenance window. With four per site, everything stays home during a rolling upgrade. That is the real difference between the minimum and the recommended number, and it is why I use four per site in both domains. I would rather spend one extra host than watch NSX Managers hop between sites every patch cycle.
NetApp MetroCluster IP, and Why the Latency Number Rules Everything
MetroCluster mirrors storage between matched controller pairs at the two sites in real time. Each site has an all-flash HA pair serving NFS to the SAP hosts. The local HA pair covers a controller failure inside a site. MetroCluster covers the loss of the whole site. SyncMirror performs synchronous replication; the ISL carries it over dedicated 100G optics, and the Mediator at Site C is the referee that authorizes an automatic switchover when a site genuinely goes down.
I keep coming back to latency in this series, and the write path is exactly why. Follow a single write:
| Step | What happens | Where |
|---|---|---|
| 1 | ESXi host sends NFS write to local controller | Local |
| 2 | Controller commits a write to local NVRAM | Local |
| 3 | Write sent across ISL to partner controller | Inter-site, latency added |
| 4 | Partner controller commits a write to its NVRAM | Remote |
| 5 | Acknowledgment sent back across ISL | Inter-site, latency added |
| 6 | Write acknowledged to the ESXi host | Local |
Steps 3 to 5 occur over the wire between sites on every single write, before the application is told the write is complete. At sub-1ms, that penalty disappears into the noise next to NVMe flash latency. At 5ms, it does not, and HANA feels it first, because the redo log sits in the commit path of every transaction. This is the whole reason Part 1 spent so long on measuring inter-site latency before designing anything. The number is not a detail. It decides whether this architecture performs or crawls.
When a site fails, the Mediator confirms that it is a real failure, not just a network partition. If the mirrors were in sync at that moment, the surviving site takes over the storage and keeps the datastores online. From the ESXi side, it is undramatic, which is exactly what you want: the datastores stay available, and HA restarts the SAP VMs on the surviving hosts.
NFS Datastore Design: Give HANA Its Own Lanes
HANA data and log volumes behave completely differently under the hood, and that alone is the reason to keep them apart. The log sits in the commit path of every transaction and needs low latency and at least 250 MB/s of write throughput. The data volume needs at least 400 MB/s, driven by savepoints, delta merges, and the reload after a restart. Those are SAP’s production minimums, and bigger systems need more. Put HANA log traffic in the same datastore as general VM I/O, and you will get latency spikes on the log exactly when the system is busiest, and you will spend a long time working out why.
So I give each HANA system dedicated data and log datastores, and keep the general SAP VMs and the workload infrastructure separate again:
| Datastore | Content | Notes |
|---|---|---|
| DS_HANA_DATA_S4 | S/4HANA data volume | Dedicated per SAP TDI |
| DS_HANA_LOG_S4 | S/4HANA log volume | Dedicated per SAP TDI |
| DS_HANA_DATA_BW | BW/4HANA data volume | Dedicated |
| DS_HANA_LOG_BW | BW/4HANA log volume | Dedicated |
| DS_HANA_DATA_SM | SolMan HANA data | Dedicated |
| DS_HANA_LOG_SM | SolMan HANA log | Dedicated |
| DS_SAP_VMS | App servers, Web Dispatcher, PO, Content Server | General SAP VM storage |
| DS_INFRA | Workload vCenter, Veeam proxies | Workload domain infrastructure |
And because the management domain is off on vSAN, these MetroCluster datastores carry only SAP. No management plane contention sneaking in. That is the split-domain design paying off again.
For NFS versions, VCF’s greenfield workflow uses NFSv3 as the primary storage for the workload domain. Other versions and storage types are reachable through import and converge paths, but for SAP on NetApp, NFSv3 with the right mount options is the proven, boring, works-every-time choice, and boring is a compliment in production storage.
One requirement that is easy to miss comes from the same Broadcom vMSC guidance: when NFS is the principal storage in a stretched workload domain, the Layer 2 network for that NFS must be stretched across both sites. This is not optional, and it is not a “nice to have”. The hosts on both sides need to access the datastores over the same stretched storage network; otherwise, the failover model simply does not work. Plan that storage VLAN as stretched from day one, not as an afterthought when something fails over unexpectedly.
Before go-live, validate the storage against SAP’s HANA requirements using SAP’s own tooling. The AFF platform being HANA-certified does not mean your specific deployment is. I have seen the consequences of skipping this: performance problems that appear months later, and a support conversation that stalls because the storage was never validated in the first place. Do the validation during the build, when it is cheap, and nobody is watching production suffer.
The Third Site: One Location, Three Referees
Every two-site HA design has the same fundamental weakness, and pretending otherwise is how you end up with split-brain. If the link between Site A and Site B is lost, each side may believe the other has died. Without an independent third party to break the tie, both sides can try to be the survivor, and now you are not protecting availability; you are corrupting data.
That is Site C’s entire job, and in this design, it does it three times over. It hosts the MetroCluster Mediator, refereeing storage switchover for the workload domain. It hosts the Pacemaker QDevice, providing the quorum vote for the SAP HA clusters I will cover in Part 3. And it hosts the vSAN witness for the management domain’s stretched cluster. Three services, three different layers, one principle underneath them all: anything that spans two sites needs a referee at a third.
Putting all three in one location is not laziness; it is correct. They all need the same thing: an independent failure domain outside the two primary sites, and one well-connected third site satisfies all three at once.
Site C does not need to be a real datacenter. It needs solid independent network paths to both primary sites, a Linux host for the Mediator, a small VM for the QDevice, the vSAN witness appliance, and optionally a NetApp system as the async replication target from the ransomware chain in Part 1.
The latency here is where people over-worry and under-check at the same time. These services are far more forgiving than the metro link, but they are not equally forgiving, and the strictest one sets your limit. The ONTAP Mediator wants under-75ms RTT to each site, with jitter under 5ms and at least 20 Mbps per DR group. The vSAN witness will tolerate up to 200ms for a cluster this size. The QDevice is the most relaxed of the three. So the Mediator is your real constraint at 75ms, which is generous enough to place Site C in an entirely different region. One more NetApp requirement worth respecting: connectivity must be genuinely independent, so that a failure at one primary site cannot also cut the path between Site C and the other. A witness who dies at the site where they were meant to arbitrate is not a witness.
Network Design: Keep the Lanes Separate
A stretched design lives or dies on network discipline. Different traffic types need different lanes for performance, security, and the sanity of whoever has to troubleshoot it at 3am.
| Network | Purpose | Traffic type | Switch / path |
|---|---|---|---|
| Production | SAP end-user access | SAP GUI, HTTP/S, Web Dispatcher | Public switch, via firewall |
| Private / internal | Inter-site replication | HANA SR, Pacemaker heartbeat, SAP interfaces | Public switch, metro backbone |
| vMotion | VM live migration | vMotion between hosts, bandwidth-intensive | Public switch, dedicated VLAN |
| Storage (NFS) | Host to storage | HANA data/log, VM datastores, latency-sensitive | Dedicated storage switch, stretched L2 |
| MetroCluster ISL | Storage sync mirror | SyncMirror replication between controllers | Storage switch, 100G optics |
| vSAN | Mgmt domain storage | vSAN ESA sync between management hosts | Public switch, dedicated VLAN |
| Management | ESXi / vCenter mgmt | Host management, SDDC Manager | OOB switch |
| IPMI / iLO | Hardware OOB | Server hardware management, PDU | OOB switch, isolated |
| Backup | Veeam to Site D | Backup/replication to Object First Ootbi | Separate from production |
The principle is boring and non-negotiable: HANA storage traffic, vMotion, backup, and management do not belong in the same lane. When they collide, performance becomes unpredictable, and unpredictable performance is the hardest kind to diagnose because it is fine until it suddenly is not.
The inter-site link deserves special attention because it carries so much: MetroCluster replication, vSAN sync for the management domain, cross-site vMotion, HANA replication, cluster heartbeats, and internal SAP traffic. Size it for the combined peak, not the daytime average. Storage replication is the load I protect first. vMotion is bursty and will happily eat bandwidth during a maintenance window. HANA replication scales with the database workload. Plan for all of it, because bandwidth planning that stops at the storage layer is only half a plan.
What Sixteen Hosts Actually Buy You
| Domain | Hosts per site | Sites | Total hosts | Storage |
|---|---|---|---|---|
| Management domain | 4 | 2 | 8 | vSAN ESA |
| SAP workload domain | 4 | 2 | 8 | MetroCluster NFS |
| Total | 8 | 2 | 16 |
Plus Site C with its three referees, and Site D with Veeam and Object First Ootbi for the recovery layer.
Sixteen hosts sound like a lot, and I understand the reaction because the first thing anyone looks at in a design is the host list. But look at what those sixteen hosts are actually buying. They keep a production SAP landscape running through a full site failure, automatically, and they keep the management plane that controls it running too, so that when the worst happens, you are not locked out of your own environment. The host count is not driven by the number of SAP VMs. It is driven by separation of duties, by real site-level HA, and by the decision to keep management available precisely when you most need to be in control. If you strip hosts out to save money, you are not trimming fat; you are removing one of those guarantees. Every host in this design is upholding a promise the business requested in Part 1.
That is the honest summary of this layer. It is not glamorous. Nobody puts compute sizing and datastore layout on a conference slide. But this is the floor the entire HA design stands on, and a beautiful HANA replication setup on top of a badly sized cluster is a beautiful thing that falls over the first time a site goes dark.
Where This Leaves Us
If you take one thing from this post, let it be that the compute and storage layer is where the whole HA design is quietly won or lost. Everything in Parts 3 and 4 sits on top of what we built here, and none of it survives a badly sized cluster or a shared storage failure domain.
The decisions I care most about defending are these three. Splitting the storage, vSAN ESA for the management domain and MetroCluster NFS for the workload domain, is the one I would fight hardest to keep. It costs a little more and adds a second storage stack to learn, but it means a storage problem on one side never takes down the tools I need to fix the other. Keeping management available during a storage or site event is worth far more than the simplicity I would gain by collapsing everything onto one platform.
I will be honest about one thing, though. A stretched vSAN cluster is not something to take lightly. vSAN is sensitive to the quality of the network between the two sites, and an unstable or congested inter-site link is where stretched vSAN designs get into trouble: latency spikes, resync storms, and split-brain behavior that is painful to unpick. This is not a place for a best-effort connection shared with other traffic. In this design, the risk is significantly reduced because the two datacenters are connected via a dedicated DWDM wave, a stable, low-latency, predictable path that does not compete with other traffic for bandwidth. The stretched vSAN is only as trustworthy as the underlying network, and the wave is what makes it trustworthy here.
Sizing against the surviving site’s N+1 usable capacity, not the raw cluster total, is the second. It is the difference between a design that looks generous on paper and one that actually holds when a site goes dark. The trap is easy to fall into and expensive to discover late.
And four hosts per site, instead of three, is the small decision that pays off in every maintenance window, not just during a disaster. One extra host per domain to keep NSX Managers from hopping sites during upgrades is, in my opinion, money well spent.
None of this is glamorous. Nobody puts a datastore layout on a conference slide. But this is the floor everything else stands on, and I would rather spend the time getting the floor right than build something beautiful on top of a cluster that falls over the first time it is truly tested.
What’s Next
In Part 3: SAP HA, I move up to the layer most people think of as “the HA design”: HANA System Replication in SYNC mode across both sites, Pacemaker with fence_vmware_rest fencing via the vCenter API, and ENSA2 for the application layer. This is where infrastructure stops being infrastructure and starts making decisions about database roles and service ownership, and where the fencing configuration can quietly break everything if you get it wrong. It is my favorite part of the whole design, and the part with the sharpest edges.
In Part 4: DR, Ransomware, and Lessons Learned, I walk through the Veeam DR design at Site D, with Object First Ootbi as the immutable target, a full ransomware attack-and-recovery scenario, and the mistakes and gotchas I encountered while designing it all.
As always, if you have questions or want to argue with any of these choices, leave a comment or reach out to me on social media. Some of the best design improvements I have made came from someone telling me why I was wrong.
Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.
