As always, when I run into an issue that takes real troubleshooting before I can fix it, and the result is something others can learn from, I mark it down as a good subject for a blog post. This “NSX Distributed Firewall Memory Exhaustion at Scale” one is very much that case.
We had an issue in five transport nodes in an environment I manage: a critical NSX alarm, DFW Memory Usage Very High, with the vsip-kentries heap at 90 percent. By the end of the investigation, the real number was around 80 hosts out of roughly 500; the distributed firewall could no longer program policy changes on a significant part of the estate, and I had learned three things that are not written in any Broadcom documentation. This post is the full investigation, the remediation that worked without downtime, and the parts where I think Broadcom deserves criticism.
All host names, VM names, and figures in this post are anonymized. The command outputs, the mechanism, and the timeline logic are real. Opinions are mine.
The environment and the alarm
The platform is a stretched vMSC estate across two sites: several hundred ESXi 8.0.3 hosts across multiple clusters, NSX 4.2.3.1 with vDefend Distributed Firewall, and DFW policies published with Applied To = DFW. The alarm text reads:
The DFW Memory usage vsip-kentries on Transport node has reached 90%
which is at or above the very high threshold value of 75%.
Recommended Action: Consider re-balancing the workloads on this host to other hosts. Keep that recommended action in mind. It matters later, and not in a good way.
What the evidence showed
On each of the five hosts I collected the threshold monitors, the kernel heap statistics, and the filter state. The heap numbers were nearly identical everywhere:
[root@esx-a12:~] /bin/vsipioctl getmeminfo | grep pfrkentry
zone 8: pfrkentry objSize = 176, inUse = 16433010, numFail = 109913, totalMem = 2892209760Millions of allocated entries of 176 bytes each, roughly 2.8 GB of a 3 GB heap, on every affected host, regardless of how many VMs the host carried. One host ran a handful of VMs at 90 percent while unaffected hosts in another cluster ran three times the VM density without an alarm. Whatever this was, it was not workload scale.
The second number is the one that decides everything: numFail above 100,000 on every host, and climbing while I watched. I captured the same host twice, minutes apart, and the counter moved by roughly 25 failures per minute. The firewall module was failing memory allocations during normal operation, continuously.
The filter state completed the picture:
[root@esx-a12:~] /bin/vsipioctl getkifflags -f nic-9381663-eth0-vmware-sfw.2
PF_KIF_FLAG_GLOBAL_TABLES 0
[root@esx-a12:~] /bin/vsipioctl getaddrsets -f nic-9381663-eth0-vmware-sfw.2 -o | head
addrset 01cefc99-... {
# flags: 0x10000025 (ROOT,LOCAL,PER,ACT,ANCREF)GLOBAL_TABLES not set, and every address set flagged LOCAL, with realization times matching the host’s last reboot weeks earlier. This matches Broadcom KB 404061 (PR 3540135) exactly. The defect: when a host imports many firewall filters in a short period, the address sets inside each filter are created as LOCAL copies. The configuration agent is supposed to flip a flag afterward and convert them into shared GLOBAL tables. Under a high import rate, it loses that race; every filter keeps its own private copy of every address set, and with twenty-plus filters each duplicating thousands of IP and MAC entries, a 3 GB kernel heap is gone.
The KB describes the trigger as bulk vMotion during maintenance mode evacuation. The estate had not done a bulk evacuation in weeks. That discrepancy led to the most interesting finding of the whole case.
The trigger nobody documents: live patching
The alarms first fired shortly after we applied the latest critical ESXi patch (the most recent VMSA security release) across the estate, using ESXi live patching. Local logs had already rotated past the campaign window, so I reconstructed it from vCenter events with PowerCLI. What the events showed: every host was patched through the live patch path. Partial maintenance mode, no evacuations, zero vMotions estate-wide, and thousands of VM reconfigure events, which is what Fast Suspend Resume looks like in the event stream.
Then the correlation, per host, to the minute:
Every host crossed the alarm threshold two to four minutes after its own FSR burst. A live patch stuns and resumes every VM on the host within seconds, and each resume tears down and re-imports the VM’s firewall filter through the same path a vMotion uses. Dozens of simultaneous filter imports are exactly the import storm the KB describes, delivered by a patching method that is marketed as non-disruptive.
Broadcom engineering later confirmed this in our support case: FSR is architecturally a local vMotion, the vNIC detaches and reattaches in the new VMM world, and a live patch campaign generates the same filter import storm as a bulk evacuation. You will not find that sentence in KB 404061. The KB does not mention live patching at all.
The full causal chain in this case: a reboot-based security patching campaign weeks earlier had loaded the heaps during the VM re-import waves, the heaps never drained because conversion could not complete, and the live patch wave pushed the roughly 80 weakest hosts over the threshold in a single afternoon.
The reboot that failed in six minutes
The obvious remediation is a reboot, and we tried it. Evacuate the affected host, reboot, clean heap, done. Except it is not done. One of the rebooted hosts raised a new critical alarm six minutes after boot. When I checked it an hour later, the heap was back at 89 to 90 percent, the allocation counter was already climbing again, and the VMs had re-imported onto the host.
The mechanism is obvious in hindsight. The reboot cleared the heap, DRS saw an empty host in a full cluster, flooded it with VMs, and the import storm recreated the exact condition the reboot was meant to fix. Broadcom engineering confirmed this too: rebooting is counterproductive, because the repopulation is itself a filter import storm.
Which brings me back to the alarm’s recommended action: re-balance the workloads on this host to other hosts. For this defect, that advice is the trigger mechanism. Bulk moves in either direction make the problem worse, and the product tells you to do exactly that. An operator who follows the product’s own guidance on a wedged estate will spread the condition to healthy hosts and risk the one failure mode this defect has in reserve: a DFW vMotion failure that brings a VM’s port up disconnected under fail-closed.
The test that showed the way out
On the rebooted host, the filter flags told an unexpected story. One of the three filters I sampled had converted to GLOBAL tables. The race is per filter, and one filter had won it even during the repopulation flood. So the conversion mechanism works. It just cannot keep up with concurrent imports.
That suggested a simple test: move one small VM, alone, with nothing else in flight, to a healthy host, and check its filters on arrival.
[root@esx-c01:~] /bin/vsipioctl getkifflags -f nic-25874589-eth0-vmware-sfw.2
PF_KIF_FLAG_GLOBAL_TABLES 1
PF_KIF_FLAG_EXPORT_SELECTIVE 1Both of the VM’s filters converted, and the destination heap did not move at all, 3 percent before and 3 percent after. A converted filter shares the existing global tables instead of duplicating them, so it effectively costs nothing.
The destination host carried a second piece of evidence in its own counters: its all-time heap peak was 89 percent, timestamped during the live patch wave, and it was now at 3 percent with two months of uptime. It had been driven nearly to the wall by the same FSR storm and had fully recovered on its own. Hosts below the ceiling drain themselves as background conversion progresses. Hosts at the ceiling cannot, because the conversion itself needs heap allocations that are failing. Memory is needed to free memory, and a host at 90 percent with a climbing failure counter stays there. Weeks of flat-lined heaps on the affected hosts proved it.
The remediation: give the hosts room to breathe
The fix follows directly from the model. A wedged host does not need to be emptied or rebooted. It needs enough released heap for conversion to restart, and then it recovers on its own. Moving a few small VMs off each affected host, one vMotion at a time with 90 seconds between moves, does exactly that: each departure releases that filter’s LOCAL duplicates, each arrival converts cleanly on a healthy destination, and the wedged host regains the headroom it needs to drain itself.
We scripted it in PowerCLI with hard rules: solo vMotions only, same cluster only, destinations verified healthy and capped at a few received VMs each, cluster service VMs and NSX components excluded in code, every move logged, a reverse-order rollback script standing by, and a full stop on any vMotion failure. For an estate this size, that is a few hundred moves and several hours of unattended runtime. No reboots, no downtime, no maintenance windows.
What the support case confirmed
We opened a support case with the full evidence set, and the engineering answers confirmed every suspicion from the investigation. FSR is architecturally a local vMotion, and a live patch campaign is a filter import storm. Host reboots are counterproductive because DRS repopulation recreates the race. The MAC set optimization workaround listed in the KB does not apply to environments with Layer 2 rules in the DFW, which describes most real environments I have seen. And the paced migration approach we devised was assessed as fully supported, safe, and aligned with engineering’s own recommendation to rate-limit the import burst.
The mechanism explanation that came back is worth repeating, because it makes the memory arithmetic obvious. During a burst, a host imports around a hundred active filters, each containing roughly a hundred address sets flagged LOCAL, and a single address set in a large security configuration can hold tens of thousands of IP and MAC entries. Every one of those addresses is written into the heap as a local table when the VM lands. The heap reaches its ceiling instantly, and at that point the configuration agent no longer has the memory overhead it needs to process the configuration push and set the GLOBAL_TABLES flag. The tables are left stranded as LOCAL permanently. Pacing the imports keeps the temporary spike below the ceiling and gives the agent the headroom to sweep the tables and convert them.
Two answers changed how we operate beyond this one estate. The race and the heap limit are strictly per destination host, since each transport node runs its own heap and its own configuration agent, so parallel migrations are safe as long as each simultaneous move targets a different destination. That makes future evacuations practical again: spread wide instead of serializing, and the only rule is never to send two VMs to the same host at once. There is no recommended maximum number of concurrent imports per destination, because the heap cost depends entirely on the size of the address sets applied to the migrating VM.
The second answer is the one to check your own estate against. The vsip architectural changes that introduce this behavior were made in the NSX 4.x branch, so 4.0, 4.1 and 4.2.x are all exposed when subjected to high-concurrency vMotions or FSR with large dynamic address sets. The exposure covers the whole 4.x branch, and it persists there until the algorithmic redesign ships in a future major version. There is no backport planned.
The result
The recovery is measurable. One host that had been pinned at 90 percent for weeks, with allocation failures accumulating at roughly 25 per minute, looked like this a few hours after a handful of VMs were paced off it:
vsip-kentries 3 104 MB 3070 MB PeakEver 91
zone 8: pfrkentry inUse = 621117, numFail = 979, totalMem = 109316592From 2711 MB to 104 MB. Objects in use down from over 16 million to 621,000. Total allocated memory returned to the system rather than merely marked free. Allocation failures went from 25 per minute to 71 in three and a half hours, which is effectively stopped. Every filter on the host now reports GLOBAL_TABLES set, including two VMs that arrived after the campaign started and converted cleanly on landing.
Counting the IP entries inside a converted filter’s address sets returns zero, which surprised me until I understood why. A converted filter holds no private copies; it references the shared global tables, and that is where the 2.6 GB went.
No reboot. No downtime. No maintenance window.
What this means for the KB
Broadcom is updating KB 404061 to include the paced single-VM drain as the official remediation path for hosts already in the exhausted state, and to document ESXi live patching and FSR as a trigger alongside bulk vMotion. Both were missing, and both matter: the first because an affected customer currently finds an article with no way to recover a wedged host, the second because the patching method most operators consider the safe one is a trigger nobody warned them about.
That is the right outcome, and credit to the support engineer who took the evidence seriously and pushed it through rather than closing the case with a workaround that does not apply. It should not have required a customer to derive the remediation from first principles, and it is fair to ask why an alarm that fires on this condition still recommends rebalancing workloads, which is the trigger mechanism. But the documentation is being fixed, which is more than many known-issue articles get.
If you run distributed firewall on any NSX 4.x release, the two commands that tell you where you stand take thirty seconds:
nsxcli -c "get firewall thresholds"
/bin/vsipioctl getmeminfo | grep pfrkentryLook at the kentries row, and look at the PeakEver column beside it. Ours recorded hosts that had been driven to 89 percent by patch campaigns nobody connected to the firewall, months of near misses sitting in a counter that nothing reads.
How to check your own hosts
Everything below runs over SSH on the ESXi transport node. Nothing here changes anything, so it is safe on production.
Step 1: find out which hosts are affected
The alarm appears in the NSX UI under Home, Monitoring, Alarms, but the UI list is paged and filtered, and I found it easy to miss hosts that way. Pull the full list from the API instead:
GET https:///api/v1/alarms?feature_name=distributed_firewall&page_size=1000 Two fields matter in each result: `node_display_name` tells you the host, and `_create_time` tells you when it first crossed the threshold. Convert those timestamps and sort them. If several hosts crossed within the same few hours, you are looking at a single triggering event rather than gradual growth, and that event is worth identifying.
One caution learned the hard way: this is a point-in-time list. An alarm that resolves, for example because someone rebooted the host, disappears from it even though the host refills minutes later. Do not treat the export as a complete picture of unhealthy hosts.
Step 2: read the threshold table on a host
nsxcli -c "get firewall thresholds"This prints eighteen rows, one per firewall memory pool. The row that matters is `vsip-kentries`. A typical wedged host looks like this:
# Name Raised Threshold CurrValue CurrSize MaxSize PeakEver EverTime(ago)
11 vsip-fprules False 90 55 1701 MB 3070 MB 56 4d 18:04:43
14 vsip-kentries False 90 90 2765 MB 3070 MB 91 4d 18:04:43Reading the columns:
`CurrValue` is the current usage as a percentage. `CurrSize` and `MaxSize` are the same thing in megabytes. The heap ceiling is 3070 MB, so 90 percent is roughly 2.7 GB consumed by firewall address tables on a single host.
`Threshold` is the value at which the row flips `Raised` to True. Note that the NSX alarm fires at 75 percent, which is well before this row reports a problem. A host at 80 percent is already alarming in NSX while this table still shows `Raised False`.
`PeakEver` and `EverTime(ago)` are the two most useful columns and the ones nobody looks at. They record the highest value the pool has ever reached and how long ago. A host sitting comfortably at 3 percent with a `PeakEver` of 89 four days ago is telling you that something drove it almost to the ceiling recently and it recovered. That is your early warning, and it is also how you identify the triggering event: line up the `EverTime` values across several hosts and you have a timeline.
Also watch `vsip-fprules` on the same row set. It moves with `vsip-kentries` during an import storm, and seeing both spike at the same timestamp is a good confirmation that the two are being driven by the same event.
Step 3: confirm whether the host is actually wedged
A high percentage on its own is not proof of the defect. This is the command that decides it:
/bin/vsipioctl getmeminfo | grep pfrkentry
zone 8: pfrkentry maxObj = -1, objSize = 176, alloc = 126481761902,
free = 126465328892, inUse = 16433010, numFail = 109913, totalMem = 2892209760What to read here:
`inUse` is the number of live address entries, and `objSize` is 176 bytes each. Multiply them, and you have the heap consumption, which should match the `CurrSize` from step 2. In the output above, 16.4 million entries is 2.8 GB.
`numFail` is the one that matters. It counts allocations that failed because the heap had no room. On a healthy host, it is zero. Any non-zero value means the firewall module is unable to allocate the memory it is asking for.
Run the same command again a few minutes later and compare `numFail`. If it has increased, the host is failing allocations right now, during normal operation, and not merely holding a high value from an earlier event. On the hosts in this case, the counter moved by 13 to 25 per minute. A static `numFail` with a falling percentage means the host is recovering on its own and needs nothing from you.
`totalMem` is worth noting too. After recovery it drops with the object count, which confirms the memory is genuinely returned to the system rather than retained and marked free internally.
Step 4: prove the cause, LOCAL address sets
List the firewall filters on the host. Each running VM contributes one filter per vNIC:
summarize-dvfilter | grep "name: nic"Ignore the `ESXi-Firewall.0` entries; those belong to the host management firewall. The ones ending in `vmware-sfw.2` are the VM filters. To find which VM owns a filter, or the filters belonging to a VM:
summarize-dvfilter | grep -A4 "" Then check the conversion state of a filter:
/bin/vsipioctl getkifflags -f nic-9381663-eth0-vmware-sfw.2Two lines in that output matter. `PF_KIF_FLAG_GLOBAL_TABLES` is the whole diagnosis: 1 means this filter shares the global address tables and costs almost nothing, 0 means it is holding its own private copy of every address set and is part of the problem.
`PF_KIF_FLAG_L2_RULES_PRESENT` decides whether the workaround published in KB 404061 is available to you. The MAC set optimization setting requires that Layer 2 rules are not in use. If this flag is 1, that workaround is off the table for your environment.
Confirm it in the address sets themselves:
/bin/vsipioctl getaddrsets -f nic-9381663-eth0-vmware-sfw.2 -o | head -20
addrset 01cefc99-f82d-4ff3-b528-f698c696f860 {
# realization time : 2026-07-11T11:23:45
# refs: 1, 0 flags: 0x10000025 (ROOT,LOCAL,PER,ACT,ANCREF)
ip 10.20.126.3,
}The `LOCAL` in the flags is the defect. On a converted filter, these are shared, and the per-filter copies do not exist. The `realization time` is useful evidence: if it matches the last host boot or the last maintenance event, the tables have been stranded since then.
To see the scale of what is being duplicated:
/bin/vsipioctl getaddrsets -f -n
/bin/vsipioctl getaddrsets -f -o | grep -c "ip "
/bin/vsipioctl getaddrsets -f -o | grep -c "mac " The `-n` form prints ACTIVE/ROOT and ACTIVE/ALL counts, which is the number of address sets attached to that one filter. The grep counts give the number of addresses inside them. Multiply address sets by addresses by filters, and the 2.7 GB stops being surprising.
A converted filter returns zero for those grep counts, which confirms it is referencing the shared tables.
Step 5: check every filter on the host at once
Sampling three filters told me the race is per filter, so a single sample can mislead. This loop reports the state of all of them:
for f in $(summarize-dvfilter | grep "name: nic" | grep vmware-sfw | awk '{print $2}' | sort -u); do
g=$(/bin/vsipioctl getkifflags -f $f 2>/dev/null | grep GLOBAL_TABLES | awk '{print $2}')
echo "$f GLOBAL_TABLES=$g"
doneAll ones means the host is healthy regardless of what the percentage was earlier. A mixture means conversion is in progress. All zeros on a host at 90 percent means it is wedged and will stay that way.
Step 6: check whether it has already caused damage
Two log searches, on the same host:
grep dfw_vmotion_failure /var/run/log/nsx-syslog.log
zcat /var/run/log/nsx-syslog.*.gz | grep -c dfw_vmotion_failure
grep pfr_attach_table /var/run/log/vmkernel.logThe first two look for the failure mode where a VM arrives on a host and its firewall filter cannot be imported, which under fail-closed brings the port up disconnected. The third looks for the kernel side of a failed filter import. Empty results mean the heap is exhausted, but nothing has broken yet, which is the moment you want to catch this in.
Note the retention. These logs rotate quickly on a busy host, and in this case they had already rolled past the triggering event, which is why the timeline had to be rebuilt from vCenter events instead.
Step 7: Check the control plane
In the NSX UI, go to Security, Distributed Firewall, and look at the state column on your policies. Realizing a published rule set requires heap allocations on every transport node in the policy span. If hosts are at the ceiling, those allocations fail, and the policies show Failed. That is the point where a memory problem has become a security problem, because rule changes are no longer reaching the data plane on those hosts.
How to recover a wedged host
You are releasing enough heap for the configuration agent to start converting the remaining filters. Once that starts, the host finishes the job itself, and it never has to be emptied.
- Pick destinations first. They must be in the same cluster, they must not be on the alarm list, and they must have real headroom. Check each candidate with `
nsxcli -c "get firewall thresholds"` and use hosts in the low tens of percent or below. Sending VMs to a host at 85 percent moves the problem rather than solving it. - Migrate one VM at a time. Choose small ones; they move fastest, and VM size has nothing to do with the benefit. Leave a pause between migrations; we used 90 seconds. The pause is the entire mechanism: it gives the configuration agent time and memory headroom to convert the newly imported filter before the next one arrives.
- Spread the destinations. Do not send several VMs to the same host in the same run. The race is per destination host, so a single destination receiving three imports in quick succession can lose it, which is how you create a new wedged host while fixing an old one.
- Verify after two or three departures. Run the threshold command and the `
getmeminfo` command again on the source. You are looking for `CurrValue` falling and `numFail` no longer increasing. On the host in this case, three departures were enough, and it went from 88 percent to 3 percent over the following hours as conversion caught up. - Do not migrate the VMs back afterward. Moving them back in a group is another import storm on a host you have just recovered.
- Never do this with maintenance mode. Putting the host into maintenance mode evacuates everything at once, which is the trigger condition, and on a wedged host it also risks the vMotion failure mode from step 6 above.
Things to avoid entirely: rebooting the host, following the alarm’s advice to rebalance workloads, running a live patch campaign, and any automated bulk evacuation. All four are import storms.
If you are scripting this across many hosts, the parts worth building in are a dry run that prints the plan before anything moves, an exclusion list for cluster service VMs and NSX appliances, a per-destination cap, a log of every move, and a stop on the first failure. A rollback script that reverses the moves in order costs an hour to write and is what lets you start the run with confidence.
Final thoughts
What stayed with me from this case is how quiet the failure was. Nothing crashed, no VM lost its network, no monitoring system escalated anything beyond a memory alarm that looks like a capacity warning. Meanwhile, the distributed firewall on a large part of the estate had stopped accepting policy changes, and the environment would have carried on that way indefinitely if nobody had opened the alarm and looked at what was underneath it. A security control that silently stops applying changes is a worse condition than one that fails loudly, and there is nothing in the alarm text that tells you that is what you are looking at.
The second thing is the position this leaves NSX 4.x operators in. The defect is confirmed, the trigger is a patching method most of us treat as the safe option, the fix requires an architectural redesign that is only coming in a future major version, and the remediation is an operational procedure rather than a code change. That is a workable answer for one environment where you can script a paced migration and watch the counters. It is a harder answer for anyone running distributed firewall at scale with large dynamic groups, because it turns routine maintenance into something that needs planning, concurrency control, and verification every time. If you are on 4.x with a heavy DFW configuration, this belongs in your upgrade reasoning, not just in your patching procedure.
I would also look at the security configuration before assuming the platform is the whole problem. This environment reached the ceiling so quickly because policies are applied to the entire distributed firewall rather than scoped to groups, so every rule and its full address list lands on every filter on every host. The defect is real, and it is Broadcom’s to fix, but the size of the address sets is what decides whether an import storm costs you a few megabytes or three gigabytes. Scoping policies properly and using ranges or dynamic criteria instead of enumerated addresses is worth doing on its own merits, and in an environment like this one it is the difference between a near miss and 80 wedged hosts.
The last point is about the counters. Everything needed to predict this was already on the hosts, in a threshold table nobody reads until an alarm fires. The PeakEver column had been recording near misses for months. If you take one thing from this post, make it a habit to check that table before and after every maintenance activity, and keep the numbers. Ours turned out to be the difference between an unexplained alarm and a documented root cause with a timeline.
Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.
