VM Network Port Attachment Failure and High CPU Usage on OVNDB

Problem

PCD environment is experiencing a critical failure in VM network port attachment operations, preventing virtual machines from connecting to the network. The service is consuming 88-95% CPU, causing it to become unavailable for legitimate API requests.

|poll_loop|INFO|Dropped 79 log messages in last 94 seconds (most recently, 91 seconds ago) due to excessive rate

|poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage)

nova.exception.PortInUse: Port <PORT_UUID> is still in use.

Environment

• Private Cloud Director Virtualization - v2025.4 and Higher
• Private Cloud Director Kubernetes – v2025.4 and Higher
• Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
• Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher

Cause

The OVN Database (OVN-DB) service has entered a non-responsive state characterised by a high-CPU spin loop.

Diagnostics

1. VM deployment fails with the below exception shown on the pf9-ostackhost logs located on the hypervisor at location /var/log/pf9/.

nova.exception.PortInUse: Port <PORT_UUID> is still in use.

2. Review the ovn-ovsdb-sb-0 pod logs from the workload region namespace.

$ kubectl logs -n <WORKLOAD_NAMESPACE> <OVN_OVSDB_POD_NAME>

3. Validate the above pod logs and check CPU usage messages like shown in below snippet.

|poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage)

Resolution

1. Take the backup of OVNDB database.

$ kubectl cp <WORKLOAD_NAMESPACE>/<OVN_OVSDB_NB_POD_NAME>:/var/lib/ovn/ovnnb_db.db  ./ovnnb_db_backup.db

$ kubectl cp <WORKLOAD_NAMESPACE>/<OVN_OVSDB_SB_POD_NAME>:/var/lib/ovn/ovnsb_db.db ./ovnsb_db_backup.db

2. Once backup is taken, execute compaction command

$ kubectl exec <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/compact

$ kubectl exec <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/compact

3. Enable memory compaction on the below pods

$ kubectl exec <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- \

  ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/memory-trim-on-compaction on

​

$ kubectl exec <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- \

  ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/memory-trim-on-compaction on

4. Restart the below pods

$ kubectl delete <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE>

$ kubectl delete <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE>

Validation

1. Check the pods' status; it should be running

$ kubectl get pods -n <WORKLOAD_NAMESPACE>  | grep -i ovn

2. Review the ovn-ovsdb pod logs from the workload region namespace and confirm if no high CPU usage messages are logged in these pods.

$ kubectl logs -n <WORKLOAD_NAMESPACE> <OVN_OVSDB_POD_NAME>

3. Deploy a new VM and review the pf9-ostackhost logs from the hypervisor (host); it no longer logs Port <PORT_UUID> is still in use messages.