Migrating Workloads to Azure VMware Solution with VMware HCX: A Practical Guide < ProVirtualzone

Based on my hands-on experience with VMware HCX, I recently spent several weeks researching Azure VMware Solution (AVS) as part of planning for a possible project with a financial services company. The customer has an on-premises VMware environment with 500 VMs across multiple clusters that they want to migrate to AVS, driven by the goal of evacuating their data center without refactoring critical applications. This blog post, “Migrating Workloads to Azure VMware Solution with VMware HCX: A Practical Guide,” outlines a practical and detailed plan based on my findings, proposing my approach for this type of migration. I won’t walk through button clicks; I will focus on strategy, architecture, tools, lessons learned from my research, and pitfalls to avoid. This is written for anyone planning to migrate workloads from on-premises to Azure VMware Solution using VMware HCX.

Let’s begin by grounding ourselves in the fundamentals of AVS and HCX to set the stage for the migration plan.

Understanding Azure VMware Solution and VMware HCX

Before discussing the migration plan, it’s important to clearly understand what AVS and HCX offer. Many professionals underestimate the nuances involved, leading to misconceptions and missteps during migration planning.

AVS is a Microsoft-managed service that provides a dedicated VMware Software-Defined Data Center (SDDC) environment hosted in Azure. It includes the familiar components: vSphere for compute, vSAN for storage, and NSX-T for network virtualization. This environment is hosted on bare-metal servers and allows integration with native Azure services like Azure SQL or Azure AD. Since AVS mirrors the on-premises VMware environment, it’s a potential option for the customer to extend or evacuate their data center without re-architecting workloads.

VMware HCX is the engine for migration. It offers tools for bulk VM movement, live migration, disaster recovery, and network extension, with features like WAN optimization (compression and deduplication) to reduce bandwidth usage. HCX is especially useful in AVS projects because it can move workloads without changing their IP addresses or reconfiguring complex dependencies. Here are the key HCX migration methods I plan to use:

vMotion: Live migration of powered-on VMs with no downtime. It’s bandwidth-sensitive, requiring careful planning.
Bulk Migration: Scheduled replication with minimal cutover downtime (~minutes). Ideal for large volumes.
Replication Assisted vMotion (RAV): Combines pre-seeding with vMotion to support large, high-change-rate VMs like databases.
Cold Migration: For VMs that can be powered off before migration, it is often used for legacy or test workloads.
OS Assisted Migration (OSAM): Used to migrate physical servers or Hyper-V workloads into vSphere.

Choosing the correct migration type is crucial. I recommend a mix of vMotion for critical workloads, Bulk for general VMs, and OSAM only when necessary, such as for non-vSphere environments. For quick reference, here’s a comparison:

Understanding these technologies upfront helps avoid unnecessary risks and downtime. It’s like knowing your tools before starting a complex project—it sets you up for success. With this foundation in place, the next step is to assess the customer’s environment thoroughly to ensure readiness for the migration.

Pre-Migration Assessment

A proper discovery phase is the foundation of every successful AVS migration. Based on my research, skipping or rushing this step can lead to significant issues, so I recommend dedicating ample time to analyzing the customer’s environment using a mix of VMware-native and Azure-native tools to ensure readiness.

Here’s how I propose we approach the assessment:

VM Inventory and Sizing: Use RVTools or vRealize Operations to collect CPU, RAM, disk, and snapshot information. Azure Migrate can generate size recommendations and flag unsupported configurations, such as legacy OS versions.
Application Dependency Mapping: Leverage Azure Migrate’s dependency visualization, running it for at least 30 days to capture traffic flows. This can reveal hidden dependencies, such as a legacy web server making backend calls to an undocumented MSSQL instance.
Compatibility Checks: Confirm vSphere version (6.5+), virtual hardware compatibility, OS support, and network topology. Use the VMware Interoperability Matrix and HCX documentation for validation.
Workload Classification: Organize workloads into categories such as mission-critical, production, development, or archival. This helps plan phased migrations and rollback options.
Special Workloads: Identify VMs requiring additional planning, such as those with raw device mappings (RDMs), PCI passthrough devices, or large disks exceeding HCX replication limits (typically 8 TB). Plan to convert RDMs to VMDKs to avoid issues.
Compliance Assessment: To ensure regulatory alignment, map requirements like GDPR or HIPAA to Azure policies.

A thorough pre-migration assessment is like laying the groundwork for a house—if we get this right, the rest of the migration becomes far more manageable. It’s all about identifying potential roadblocks early. With the assessment planned, let’s move on to preparing the on-premises environment for HCX integration.

Preparing the On-Premises Environment

Once the discovery phase is complete, the next step is preparing the customer’s on-premises infrastructure for HCX integration. This phase is mostly technical, but I recommend coordinating with network and firewall teams early to avoid delays derailing the timeline.

Here’s the preparation plan I propose:

HCX Connector Deployment: Deploy the HCX Connector OVA in the vSphere environment. Register it with vCenter and NSX Manager, and confirm successful registration before proceeding to the service mesh setup.
Network Configuration and Readiness:
- Validate Layer 3 connectivity between the on-premises site and the AVS environment.
- Open required ports (TCP 443, UDP 4500) between all HCX appliances.
- Ensure DNS resolution for vCenter, HCX, and NSX components in both directions.
- Synchronize NTP settings to avoid time drift issues affecting authentication or replication.
Grouping Workloads: Create logical migration groups based on applications, dependencies, and business windows. This simplifies wave planning and testing, such as grouping web servers and databases together.

Thorough preparation of the on-premises environment sets the tone for the entire migration. A small oversight, like a time sync issue, can lead to hours of troubleshooting, so attention to detail here is critical. Now that the environment is ready, let’s move on to strategic planning, where we’ll design the migration architecture.

Strategic Planning for VMware to AVS Migration

Strategic planning is where the migration starts to take shape. This phase requires careful consideration of networking, resource sizing, HCX configurations, and AVS setup. I’ll break this down into key areas to ensure we’re set up for success.

Networking Considerations

This is the most complex part of the architecture. Networking in HCX and AVS is powerful but unforgiving if misconfigured, so let’s dive into the details tailored to the customer’s potential needs.

ExpressRoute: This is recommended for production workloads requiring low latency and high bandwidth (e.g., financial apps with real-time data). Use private peering and ensure MTU settings support 1600 bytes for HCX performance. It is suitable if the customer has a large-scale deployment or multi-site connectivity needs.
Site-to-Site VPN is a cost-effective alternative for smaller or test setups with higher latency tolerance. It is ideal if the customer’s initial migration is limited to a pilot phase or non-critical workloads.
Decision Matrix: Use ExpressRoute for >100 VMs or high-performance requirements; VPN for <50 VMs or initial testing.
Global Reach: This is required when ExpressRoute circuits span multiple Azure regions (e.g., the customer operates across the East and West US). It enables routing between hubs and AVS private clouds—consider this if the customer has a multi-region strategy.
Layer 2 Network Extension: HCX NE allows VMs to retain IP addresses. Each appliance supports up to 8 VLANs—plan for additional appliances if the customer needs to extend more networks.
Mobility Optimized Networking (MON): Resolves asymmetric routing by enabling optimized egress through Azure. Configure firewall rules for return traffic, especially if the customer relies on extended networks.
Hub-Spoke Topology: Place AVS in a spoke VNet connected to a hub with shared services (DNS, identity, monitoring). It is recommended that the customer centralize security and management.
Azure vNet Integration: Peers AVS VNet with hub VNet for hybrid scenarios with Azure services, ensuring customers can leverage Azure-native tools.

Here’s a diagram of the network setup:

</p>
<p><img bv-data-src="https://www.provirtualzone.com/wp-content/uploads/2025/05/HCX-to-AVS-04.png"  loading="lazy" decoding="async" class="alignnone wp-image-20885 bv-tag-attr-replace bv-lazyload-tag-img"   src="data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20850%20670'%3E%3C/svg%3E" alt="Migrating Workloads to Azure VMware Solution with VMware HCX: A Practical Guide" width="850" height="670"/><br />

AVS Configuration

Configuring the AVS environment is critical to meet the customer’s needs. This involves setting up the SDDC to align with the on-premises environment and migration goals.

Node Sizing: Start with AV36 nodes (36 cores, 576 GB RAM, 15.36 TB NVMe) in a minimum three-node cluster. Scale based on the customer’s VM count (e.g., 500 VMs may require 6-8 nodes) and performance requirements.
vSAN Configuration: Enable Fault Tolerance (FTT=1, RAID-1) for redundancy. Plan for deduplication and compression to optimize storage, but monitor potential overhead from replication traffic and snapshots.
NSX-T Setup: Configure NSX-T segments to match the customer’s on-premises network topology. Plan for L2 extension integration and define security policies (e.g., distributed firewall rules).
Azure Integration: Set up Azure vNet peering and integrate with a hub VNet for shared services (e.g., DNS, Azure AD). Ensure compatibility with the customer’s existing Azure subscriptions.
Resource Allocation: Allocate CPU, RAM, and storage based on Azure Migrate sizing recommendations, adding a 20-30% buffer for growth.

Network Design Considerations for HCX with AVS

When migrating workloads from on-premises environments to Azure VMware Solution using VMware HCX, the network design becomes one of the most critical parts of the architecture. Even though HCX handles much of the complexity of mobility and network extension, improper routing, segmentation, and appliance placement planning can cause serious disruptions or failed migrations.

To support the concepts in this section, I created the image below summarizing the most important network design considerations when planning HCX migrations to AVS.

Hub-and-Spoke Topology

The upper-left quadrant of the diagram shows the hub-and-spoke network architecture. This is the recommended design pattern in Azure. The AVS private cloud is deployed as a spoke virtual network (VNet), while the hub contains centralized services such as DNS, Active Directory, NVA firewalls, monitoring, and ExpressRoute gateways. Each spoke connects to the hub via VNet peering, allowing traffic flow through shared services while isolating lateral movement across spokes.

This design simplifies routing, enforces security policy at a central point, and ensures consistency across regions or environments.

NSX-T Segment Planning

AVS uses NSX-T for network virtualization, and HCX can stretch Layer 2 networks from your on-premises environment into AVS. But not everything can or should be stretched.

The top-right quadrant in the image illustrates a common issue: overlapping subnets. Extending networks into AVS that use the same IP ranges as existing segments will cause routing conflicts and unpredictable behavior. The image shows how duplicating a subnet like 192.168.12.0/24 can introduce problems if it is extended from different sources or used more than once.

Always document and plan your subnets carefully before deploying Network Extensions.

Service Mesh Scope

In the lower left quadrant, we cover HCX Service Mesh considerations. This is the core of HCX connectivity—it defines the path between your on-premises and AVS environments. Each Service Mesh includes appliances like Interconnect and Network Extension deployed on both sides.

Key considerations:

Each NE appliance supports up to 8 extended networks
You must consider throughput limits and expected migration concurrency
Service Meshes must be scoped carefully per cluster

The diagram reinforces that NE appliances can hit scale limits. When extending more than eight networks, deploy additional appliances or split migrations across multiple meshes.

Mobility Optimized Networking (MON)

This is one of the most overlooked yet critical features when using Layer 2 Network Extension. As shown at the center of the image, Mobility Optimized Networking (MON) allows migrated VMs to use local routing in AVS, even though they retain their original on-premises IP addresses.

Without MON, return traffic from the internet or Azure services would be routed back through the on-premises gateway, leading to asymmetric routing and possible traffic loss. The diagram shows two possible flows:

A dashed HCX MON path, where the VM uses the Azure gateway directly
A curved path marked Asymmetric Route, which is the non-optimized path leading to routing problems

Enabling MON requires:

That your AVS segments have NSX-T gateway support
Correct firewall rules to allow local egress
Awareness of routing changes and test validation

MON solves many post-migration problems, especially for applications that access Azure-native services or internet endpoints.

Firewall and Routing

Another core point shown in the upper-right quadrant is the importance of firewall and routing awareness. HCX requires several ports (like TCP 443 and UDP 4500) to be open bidirectionally between all appliances. Return routing must be predictable and symmetric, especially when MON is enabled.

If you use network virtual appliances (NVAs) in Azure or on-premises, ensure they do not perform NAT or block traffic HCX uses. When connectivity issues appear, they are often traced back to firewall or routing misconfigurations.

ExpressRoute and MTU Considerations

Lastly, the bottom-right quadrant highlights something many teams miss during validation: MTU size and ExpressRoute configuration. HCX Interconnect traffic is sensitive to fragmentation. AVS environments require a minimum MTU of 1500, and enabling jumbo frames (1600 bytes or higher) is highly recommended for performance and reliability, especially when using Bulk or RAV migration methods.

Always test MTU paths end-to-end before starting production migrations.

Cluster and Resource Planning

The AVS private cloud should be based on the customer’s real usage data, not just the number of on-premises hosts. AVS nodes offer consistent performance, but consolidation requires careful planning.

Capacity Planning: Use Azure Migrate or vRealize Operations for CPU, RAM, and storage usage reports. Add a 20-30% buffer for growth and spikes.
Cluster Sizing: Start with a minimum three-node cluster and scale incrementally based on the 500 VM load.
Storage Considerations: Configure vSAN with FTT=1, RAID-1 for redundancy. Monitor deduplication/compression to manage storage usage.

HCX Service Mesh and Migration Methods

The HCX Service Mesh defines how migration traffic flows, making it the cornerstone of migration infrastructure. Misconfigurations here can derail the project.

Deploying Service Mesh: Pair the on-premises HCX Connector with the AVS HCX Cloud Manager. Define compute and network profiles for accurate routing.
HCX Interconnect: Handles secure, encrypted migration traffic. Ensure adequate bandwidth.
Network Extension (NE): Required for L2 stretching; plan capacity (8 VLANs per appliance).
Mobility Optimized Networking (MON): Prevents asymmetric routing post-migration.
Migration Methods:
- vMotion: Critical apps (e.g., web servers).
- Bulk: Dev/test VMs.
- RAV: Databases (e.g., SQL Server).
- Cold: Legacy systems.
- OSAM: Physical Linux servers.

Here’s the Service Mesh layout:

Getting the networking, AVS configuration, sizing, and HCX configurations right during planning is crucial; it’s like building a solid foundation for a house. A small mistake, like an overlapping subnet, can cause significant delays. With the architecture designed, let’s move on to planning the migration execution.

Migration Execution and Operations

Executing the migration effectively involves careful planning and rigorous validation. This is where the preparation pays off, and I’ll outline a methodical approach to ensure success.

Here’s the execution plan I propose:

HCX Setup: Pair on-premises and AVS vCenters using HCX Manager. Deploy Service Mesh with compute/network profiles.
Network Extensions: Extend VLANs via HCX NE; validate with ping tests to catch issues early.
Pilot Wave: Start with 5-10 non-critical VMs using Bulk Migration to test the process.
Scheduled Waves: Align with business windows. Use vMotion/RAV for critical VMs to minimize downtime.
Rollback Strategy: Retain powered-off source VMs until validated.
Monitoring and Logging: Use HCX dashboards; tag VMs for tracking to maintain visibility.

Here’s the migration workflow:

The key to success is a phased approach with rigorous validation at each step. The pilot wave will help us identify and resolve issues before the production migration. Once the workloads are in AVS, the next phase is planning for optimization and finalization.

Post-Migration Optimization and Finalization

Post-migration optimization is often overlooked but is critical to fully leveraging the benefits of moving workloads to AVS. This phase ensures optimal performance, resource efficiency, and cost control.

Here’s the optimization plan I recommend:

Performance Validation: Use Azure Monitor and vRealize Operations to check CPU, memory, disk latency, and network throughput.
Validation Checklist:
- App functionality.
- DNS resolution.
- Network connectivity.
Right-Sizing Workloads: Oversized VMs waste resources. Use Azure’s recommendations to downsize where possible.
Decommissioning Network Extensions: Remove unused L2 extensions after transitioning to NSX-T segments.
Cost Management: Use Azure Cost Management to set budgets and alerts. Schedule shutdowns for non-production VMs during off-hours.
Governance:

Post-migration optimization maximizes the investment in AVS and ensures long-term stability. It’s not just about getting to the cloud—it’s about running efficiently once there. With the environment optimized, let’s address the critical aspects of security, compliance, and governance.

Security, Compliance, and Governance

Migrating workloads to AVS involves integrating the infrastructure with Azure’s governance and security model. To protect workloads and maintain trust, it’s essential to extend organizational standards into the AVS environment seamlessly.

Here’s the security and governance plan I propose:

Identity and Access Management (IAM): Integrate Azure Active Directory (Azure AD) for role-based access control (RBAC). Restrict vCenter access to necessary personnel.
Security Integration: Use Azure Firewall, Defender for Cloud, and Azure Sentinel for threat detection and response.
Compliance Management: Leverage Azure Policy to enforce standards like PCI-DSS, GDPR, or ISO 27001. Conduct regular audits with Azure Security Center.
Centralized Logging and Monitoring: Stream logs to Azure Monitor or Sentinel for better visibility.

Strong security, compliance, and governance measures are essential for protecting workloads and ensuring business continuity, especially for regulated industries like finance. Let’s share some final thoughts and reflections on the migration plan.

Final Thoughts

Planning a migration to AVS using VMware HCX is a strategic, detail-oriented task that requires careful preparation and deep VMware expertise. Based on my research, a successful migration hinges on accurate assessments, detailed preparation, robust networking configurations, proper AVS setup, and thorough post-migration optimization. This plan for moving the customer’s 500 VMs outlines a methodical approach to achieve a seamless transition with minimal disruption.

While the technology and tools available through HCX and AVS make the process more manageable, I’ve learned never to underestimate the importance of clear communication, detailed documentation, and rigorous validation at every step. Aligning migration windows with business needs and ensuring well-coordinated teams are as crucial as the technical setup. I highly recommend investing significant effort in planning and testing upfront—it reduces risks. It ensures a smoother transition, making the migration a business success rather than just a technical achievement.

I hope my findings and this plan help you on your AVS migration journey. If you have questions or experiences you’d like to share, feel free to comment or reach out directly—I look forward to engaging in further discussions.

FAQ

Here are answers to common questions I’ve encountered:

What downtime should I expect? It varies: vMotion offers none, Bulk Migration takes a few minutes, and Cold Migration involves longer downtime.
How does licensing work with AVS? Azure Hybrid Benefit can be used to reduce costs for Windows Server and SQL Server licenses.
What if something goes wrong? For a safe rollback, retain source VMs until the migrated workloads are fully validated.

Comments

Leave a Reply Cancel reply