VMs Inaccessible due to Underlying Hypervisors Frequently Going Offline

Problem

The VMs are inaccessible via SSH and console as the Hypervisor connectivity is flapping frequently.

$ ssh ubuntu@<VM_IP> Connection closed by [VM_IP] port 22

Environment

Private Cloud Director Virtualization - v2025.4 and Higher
Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
Component - Networking

Cause

When the DHCP lease expires, the host interface gets reconfigured. During this process, interfaces that are attached to the bridge also receive IP addresses unexpectedly. Because of this, ovn-controller enters a failed state while processing its configuration updates.
This is a known issue and is reported internally with ID: PCD-2492.

Diagnostics

In the Neutron logs:

pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via [IP] dev br-phy1 metric 0 failed. 2:b'RTNETLINK answers: File exists\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 625, in handle_msg current_config = get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 262, in get_current_config config = app_db.get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/app_db.py", line 80, in get_current_config 'config': app.get_config(), File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app.py", line 311, in get_config raise ConfigOperationError() pf9app.exceptions.ConfigOperationError

Network related files:

$ ls -lt /run/systemd/network/ -rw-r----- 1 root systemd-network 213 Jun 11 06:25 10-netplan-bond0.600.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.601.netdev -rw-r----- 1 root systemd-network 130 Jun 11 06:25 10-netplan-bond0.601.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.602.netdev -rw-r----- 1 root systemd-network 131 Jun 11 06:25 10-netplan-bond0.602.network -rw-r----- 1 root systemd-network 65 Jun 11 06:25 10-netplan-bond0.3001.netdev -rw-r----- 1 root systemd-network 155 Jun 11 06:25 10-netplan-bond0.3001.network -rw-r----- 1 root systemd-network 63 Jun 11 06:25 10-netplan-bond0.600.netdev -rw-r----- 1 root systemd-network 196 Jun 11 06:25 10-netplan-bond0.netdev -rw-r----- 1 root systemd-network 195 Jun 11 06:25 10-netplan-bond0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f0.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f0.network -rw-r----- 1 root root 82 Jun 11 06:25 10-netplan-enp1s0f1.link -rw-r----- 1 root systemd-network 134 Jun 11 06:25 10-netplan-enp1s0f1.network

Hostagent Logs:

pf9_app.py ERROR - pf9-neutron-ovn-controller:get_config failed: b'' Command: sudo ip route add default via <IP> dev br-phy12 metric 0 failed. 2:b'RTNETLINK answers: File exists\n' session.py ERROR - Bad message, app config or reading current app config. Message : {'opcode': 'heartbeat'} Traceback (most recent call last): File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 625, in handle_msg current_config = get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 262, in get_current_config config = app_db.get_current_config() File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/app_db.py", line 80, in get_current_config 'config': app.get_config(), File "/opt/pf9/hostagent/lib/python3.9/site-packages/pf9app/pf9_app.py", line 311, in get_config raise ConfigOperationError() pf9app.exceptions.ConfigOperationError

Resolution

The fix for the bug is available in the v2025.6 release.

Workaround

Check if the interface IP is assigned to the bridge and the default route is associated with the bridge.

If yes, delete the default route via bondx.x.
If not, reboot the host and check again.

$ ip r default via [VM_IP] dev br-phy12 default via [VM_IP] dev bond0.10 proto static 1.1.1.1/24 dev br-phy12 proto kernel scope link src 1.1.1.251 1.1.1.2/24 dev bond0.10 proto kernel scope link src 1.1.1.251 1.1.1.3/24 dev br-phy15 proto kernel scope link src 1.1.1.251 1.1.1.4/24 dev bond0.50 proto kernel scope link src 1.1.1.251 $ sudo ip route del default via <VM_IP> dev bond0.10

Additional Information

For further questions/concerns regarding the bug, reach out to the Platform9 Support Team.

Last updated on

Was this page helpful?