VM Network Port Attachment Failure and High CPU Usage on OVNDB

Problem

PCD environment is experiencing a critical failure in VM network port attachment operations, preventing virtual machines from connecting to the network. The service is consuming 88-95% CPU, causing it to become unavailable for legitimate API requests.

|poll_loop|INFO|Dropped 79 log messages in last 94 seconds (most recently, 91 seconds ago) due to excessive rate |poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage)
nova.exception.PortInUse: Port <PORT_UUID> is still in use.

Environment

  • Private Cloud Director Virtualization - v2025.4 and Higher

  • Private Cloud Director Kubernetes – v2025.4 and Higher

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

  • Self-Hosted Private Cloud Director Kubernetes - v2025.4 and Higher

Cause

The OVN Database (OVN-DB) service has entered a non-responsive state characterised by a high-CPU spin loop.

Diagnostics

  1. VM deployment fails with the below exception shown on the pf9-ostackhost logs located on the hypervisor at location /var/log/pf9/.

nova.exception.PortInUse: Port <PORT_UUID> is still in use.
Note:

Step 2 -3 only accessible and applicable for the Self-hosted Private Cloud Director. For SaaS environments, please reach out to Platform9 support.

  1. Review the ovn-ovsdb-sb-0 pod logs from the workload region namespace.

$ kubectl logs -n <WORKLOAD_NAMESPACE> <OVN_OVSDB_POD_NAME>

  1. Validate the above pod logs and check CPU usage messages like shown in below snippet.

|poll_loop|INFO|wakeup due to 0-ms timeout at ../ovsdb/jsonrpc-server.c:616 (95% CPU usage)

Resolution

Note:

These resolution steps are only applicable for the Self-hosted Private Cloud Director. For SAAS environment please reach out to Platform9 support.

  1. Take the backup of OVNDB database.

$ kubectl cp <WORKLOAD_NAMESPACE>/<OVN_OVSDB_NB_POD_NAME>:/var/lib/ovn/ovnnb_db.db ./ovnnb_db_backup.db $ kubectl cp <WORKLOAD_NAMESPACE>/<OVN_OVSDB_SB_POD_NAME>:/var/lib/ovn/ovnsb_db.db ./ovnsb_db_backup.db

  1. Once backup is taken, execute compaction command

$ kubectl exec <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/compact $ kubectl exec <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/compact

  1. Enable memory compaction on the below pods

$ kubectl exec <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- \ ovn-appctl -t /var/run/ovn/ovnnb_db.ctl ovsdb-server/memory-trim-on-compaction on $ kubectl exec <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE> -- \ ovn-appctl -t /var/run/ovn/ovnsb_db.ctl ovsdb-server/memory-trim-on-compaction on

  1. Restart the below pods

$ kubectl delete <OVN_OVSDB_NB_POD_NAME> -n <WORKLOAD_NAMESPACE> $ kubectl delete <OVN_OVSDB_SB_POD_NAME> -n <WORKLOAD_NAMESPACE>

Validation

Note:

Validation steps 1 & 2 are only applicable for the Self-hosted Private Cloud Director. For SAAS environment please reach out to Platform9 support.

  1. Check the pods' status; it should be running

$ kubectl get pods -n <WORKLOAD_NAMESPACE> | grep -i ovn

  1. Review the ovn-ovsdb pod logs from the workload region namespace and confirm if no high CPU usage messages are logged in these pods.

$ kubectl logs -n <WORKLOAD_NAMESPACE> <OVN_OVSDB_POD_NAME>

  1. Deploy a new VM and review the pf9-ostackhost logs from the hypervisor (host); it no longer logs Port <PORT_UUID> is still in use messages.


Next to read:
How To Reset Volume States
null
On This Page
VM Network Port Attachment Failure and High CPU Usage on OVNDBProblemEnvironmentCauseDiagnosticsResolutionValidation