How to Delete Stale Storage Host Entries in PCD

Problem

Unable to launch new VMs using self_hosted_product_name Management plane UI, Cinder-scheduler pods logs show volume service down errors.

WARNING cinder.scheduler.host_manager [None req-ID] volume service is down. (host: [HOST-UUID@STORAGE-NAME])

The volume service list has multiple volumes in down state:

$ openstack volume service list +------------------+-----------------------+------+---------+-------+------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-----------------------+------+---------+-------+------------+ | cinder-scheduler |cinder-scheduler-[UUID]| nova | enabled | down | [Timestamp] | +------------------+-----------------------+------+---------+-------+------------+

Environment

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher.

Procedure

Steps to delete the stale volumes from the self_hosted_product_name:

  1. List the storage volume services, to identify the stale storage host. Execute below command from the Host having the PersistentStorage role applied:

$ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service list
  1. Validate the above command output with below command output, capture the information about stale storage host UUID.

$ openstack volume service list
  1. After confirmation storage volume service can be deleted. Execute the below command from the PersistentStorage role host.

$ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service remove cinder-scheduler <HOST_UUID>

Validation

The stale volumes should be removed in the volumes list, It can be verified using:

  1. Export the .rc file and execute below command:

$ openstack volume service list
  1. From the PersistentStorage role host., execute the below command to ensure the stale entries are removed.

$ LD_LIBRARY_PATH="/opt/pf9/pf9-cindervolume-base/pf9-lib:/opt/pf9/python/pf9-lib:${LD_LIBRARY_PATH}" PYTHONPATH="/opt/pf9/python/lib/python3.6:/opt/pf9/pf9-cindervolume-base/lib/python3.6/site-packages:${PYTHONPATH}" /opt/pf9/pf9-cindervolume-base/bin/cinder-manage --config-dir /opt/pf9/etc/pf9-cindervolume-base/conf.d service list