In the last few weeks I had to fix several issues for some of our customers. Some of them are good examples to share, so I will pick a few and write a few simple, short blog posts for each one about what the issue was and what the fix was. This is the first of those posts. All names, IPs, and job names are dummy, but the command output, error messages, and numbers are real.
This one was a bit strange. The error pointed in one direction, the real problem was somewhere else, and there was no obvious root cause, so it needed some extra troubleshooting to find it. The short version: a silent Veeam NBD fallback, made worse by the network path.
The environment
- Veeam Backup & Replication v13 on a dedicated VBR server (Windows), which is itself a VM
- Backup proxy role on the same VM, transport mode set to Direct storage access (SAN)
- Scale-out Backup Repository with two NFS-backed extents
- Backup traffic path: proxy on subnet 192.168.40.0/24, NFS servers on 192.168.70.11 and 192.168.70.12
- vSphere cluster of 3 hosts, distributed switch for storage traffic
The symptom
Every few days, one or more backup jobs failed with a metadata desync:
Error: Cannot proceed with the job: existing backup meta file 'FILESRV01.vbm' on repository
'Scale-out Backup Repository 1' is not synchronized with the DB.
To resolve this, run repository rescanA SOBR rescan fixed it every time. And every time, it came back days later. When a fix works but the problem returns, you have not fixed anything. You have treated a symptom. So I stopped rescanning and started investigating.
The investigation
The wrong hypothesis first
My first theory was space. Both NFS extents were above 75% used, the repository uses Data Locality placement, and some of the protected VMs are large. Space exhaustion mid-write is a classic cause of corrupted metadata.
I checked it and disproved it. Free space on both extents was tight but sufficient for the running chains, and none of the failed sessions logged an out-of-space condition. A disproven hypothesis still saves the next engineer from walking the same dead end, so it stays in the notes.
Finding 1: the silent Veeam NBD fallback
The proxy is configured for Direct storage access. But the proxy is a VM, and a SAN transport proxy needs to see the storage LUNs. First check, read-only:
Get-Disk | Select-Object Number, FriendlyName, BusType, Size, OperationalStatusNumber FriendlyName BusType Size OperationalStatus
------ ------------ ------- ---- -----------------
0 VMware Virtual disk SAS 107374182400 Online
1 VMware Virtual disk SAS 2199023255552 OnlineTwo virtual SAS disks. No LUNs. This proxy has no possible path to Direct storage access. The job debug log shows what it does instead:
[ViProxyEnvironment] The proxy does not have SAN modeThe Veeam NBD fallback is documented behavior, not a bug. Failover to Network mode when the primary transport mode fails is enabled by default, and when the data cannot be processed by any other transport mode, it applies automatically even if you disable the failover option. The job completes with a Success status. No warning is raised. The transport actually used only appears in the session details, in the per-disk line, where the mode shows in brackets:
Using backup proxy veeam-proxy01 for disk Hard disk 3 [nbd]Nobody reads the per-disk lines on green jobs. The only symptom anyone notices is speed. In this case, the largest fileserver backup was processing at 23 MB/s.
A 4 TB disk at 23 MB/s means the job for VM FILESRV01 was running 6 to 10 hours every single night. That alone is not a failure. But it sets the stage for one.
Finding 2: the actual failure mechanism
The metadata desync was the aftermath, not the event. To find the event, I went to the task session logs of the first failed run and found where the chain actually broke:
Error: read: End of file [asio.misc:2]
Failed to upload disk 'multilinks>'This is a Boost.Asio TCP disconnect. The proxy was writing to the NFS repository, and mid-upload the TCP connection dropped. The session died with the write incomplete, the .vbm metadata file was left inconsistent, and every following run failed at pre-process until a rescan reconciled it.
The failure timeline confirmed it was path-level and not VM-level. On both failure events, multiple VMs in overlapping jobs dropped within 20 to 50 seconds of each other. Individual VMs do not coordinate their failures. Network paths do.
So now the question changed from “why does Veeam desync” to “what is killing long-lived TCP sessions between 192.168.40.x and 192.168.70.x”.
Finding 3: there is a router in the backup write path
Check where the write traffic actually goes, read only:
Test-NetConnection -ComputerName 192.168.70.11 -Port 2049 -InformationLevel DetailedComputerName : 192.168.70.11
RemotePort : 2049
InterfaceAlias : Storage
SourceAddress : 192.168.40.202
NextHop : 192.168.40.254
TcpTestSucceeded : TrueNextHop: 192.168.40.254. The backup writes do not stay on Layer 2. Every byte crosses a Layer 3 device between the storage subnet and the NFS subnet. Any router, firewall or L3 switch in that position can have session or idle timeout behaviour that tears down long-lived connections. A 6 to 10 hour NBD upload is exactly the kind of connection that hits those timeouts.
The slow NBD transport does not cause the drop, but it maximizes exposure to it. A backup that should take 1 hour survives a fragile path far more often than one that takes 9.
Finding 4: the jumbo frame mismatch
While mapping the path, I checked MTU end to end, because the storage network was designed for jumbo frames. The distributed switch, via PowerCLI:
Get-VDSwitch | Select-Object Name, MtuName Mtu
---- ---
Storage-dvSwitch 9000The VBR storage NIC, on the server itself:
Get-NetAdapterAdvancedProperty -Name "Storage" | Where-Object { $_.DisplayName -match "Jumbo|MTU|Packet" } | Select-Object Name, DisplayName, DisplayValueName DisplayName DisplayValue
---- ----------- ------------
Storage Jumbo Packet 1514The network is built for 9000, and the server is at 1500. Then the definitive test, a ping with the Don’t Fragment flag set at jumbo payload size (8972 bytes of payload plus 28 bytes of headers equals 9000):
ping 192.168.70.11 -f -l 8972
Packet needs to be fragmented but DF set.
ping 192.168.70.11 -f -l 1472
Reply from 192.168.70.11: bytes=1472 time<1ms TTL=63Standard frames pass, jumbo frames do not. A jumbo mismatch on a path can produce exactly this profile: small packets and short sessions work fine, large sustained NFS writes fail. Since the proxy is a VM, fixing this means the whole chain has to be aligned: the VM’s NIC, the port group, the dvSwitch, the physical uplinks, the router at .254 and the NFS server interfaces. A half-applied jumbo config is worse than none, because a 9000 NIC talking to a 1500 path makes large writes fail harder.
One warning if you go change this: applying an MTU change resets the NIC and drops any in-flight session. A dropped write mid-upload is the exact failure you are trying to fix. Confirm nothing is running first:
Get-VBRBackupSession | Where-Object { $_.State -eq "Working" } | Select-Object JobName, State, CreationTimePutting it together
The root cause is not one thing. It is a chain:
- The proxy has no LUN access, so the Veeam NBD fallback kicks in. Large backups stretch to 6 to 10 hours.
- The write path crosses an L3 device between the storage and NFS subnets, and long-lived TCP sessions across it get dropped. That drop is the
asio: End of filein the logs. - An MTU mismatch (dvSwitch at 9000, VBR NIC at 1500, path not jumbo clean) adds a second path-level factor that fits the “large sustained writes fail, small traffic passes” symptom.
- The slow runtime makes jobs overlap, so a single session drop takes concurrent VMs down together, which is why failures always came in clusters.
Not one of these is visible in the Veeam job configuration. The customer’s job settings were fine the whole time.
The remediation plan
Ordered by impact and risk:
- Confirm the full path MTU hop by hop before touching anything (physical uplinks, the device at .254, NFS interfaces). Only then align the VBR NIC to 9000, in a maintenance window, with the jumbo ping retest immediately after. If the jumbo ping fails after the change, revert to 1500 before the next scheduled job.
- Fix the transport mode. The proxy VM sits on the same cluster as the workload, so HotAdd (Virtual appliance mode) is architecturally viable. The Veeam requirements are that the proxy VM runs on a host with access to the datastores containing the disks of the VMs being backed up, and that the proxy and the processed VMs reside in the same datacenter. Confirm both, then switch. This cuts runtime dramatically, which shrinks the overlap window and the exposure to the fragile path even before the network fix lands.
- Give the network owner a concrete question, not a vague complaint: what is 192.168.40.254, and what is its session or idle timeout for NFS (TCP 2049) and long-lived flows between these two subnets? Either exempt that traffic or put VBR and NFS on the same L2 segment so writes stop crossing a router.
- As a controlled test, reduce the upload streams in the Global Network Traffic Rules. By default, Veeam transfers job data over 5 TCP/IP connections, and the option’s own description in the console says to disable it when running a large number of concurrent jobs or for networking equipment compatibility purposes. Fewer concurrent TCP connections crossing the L3 device, changed as a single variable so the effect is measurable.
The takeaways
A green job is not proof of a healthy transport mode. The Veeam NBD fallback is enabled by default and raises no warning. If you have never opened a session’s per-disk lines to check which mode appears in brackets, do it today. Direct storage access configured is not Direct storage access used.
When a rescan fixes it and it comes back, the repository is the victim, not the culprit. Metadata desync is where the damage lands, not where it starts. Go find the session where the chain first broke and read that log.
Failures that come in clusters point at paths, not at VMs. Multiple VMs dying within seconds of each other is a network event, whatever the error message says.
The whole diagnostic is read-only. Get-Disk, the session logs, Test-NetConnection, Get-VDSwitch, Get-NetAdapterAdvancedProperty and a DF-flag ping run on a production VBR without a change window. Only the fixes need change management.
Final thoughts
For me, the main lesson of this one is simple. When the same error keeps coming back, and the fix is always the same workaround, stop fixing and start investigating. The rescan was hiding the real problem for weeks.
The second lesson is to never trust the configuration screen. The proxy said Direct storage access; the reality was NBD. Checking the transport mode in the session details is now part of every Veeam health check I do, together with the jumbo ping test from end to end. Both take five minutes, and both are read-only.
And when the backup path crosses a router, ask why. Backup traffic should stay as close to Layer 2 as possible. Every extra hop is one more device with its own timeouts and its own MTU that can break a long write without leaving any trace in Veeam.
