Unable to Delete Stale Compute Service

Problem

  • Unable to delete stale compute service with the below error.

$ openstack compute service delete [COMPUTE_SERVICE_UUID] Failed to delete compute service with ID '[COMPUTE_SERVICE_UUID]': HttpException: 500: Server Error for url: https://[example.platform9.com]/nova/v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID], Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'ValueError'> 1 of 1 compute services failed to delete.

Environment

  • Private Cloud Director Virtualization - v2025.4 and Higher

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

  • Nova

Cause

  • Forcefully removing a host from PCD, resulting in incomplete cleanup of resources on the management plane.

Diagnostics

  • Compute Service List:

$ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+---------------------+---------+--------+ | ID | Binary | Host | Status | State | +---------------------------+----------------+---------------------+----------+-------+ | [COMPUTE_SERVICE_UUID_1] | nova-scheduler | nova-scheduler | enabled | up | | [COMPUTE_SERVICE_UUID_2] | nova-conductor | nova-conductor | enabled | up | | [COMPUTE_SERVICE_UUID_3] | nova-compute | [HOST1.EXAMPLE.COM] | enabled | up | +---------------------------+----------------+---------------------+----------+-------+

  • Resource Provider List:

$ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+ | uuid | name | +--------------------------------------+---------------------+ | [RESOURCE_PROVIDER_UUID_1] | [HOST1.EXAMPLE.COM] | | [RESOURCE_PROVIDER_UUID_2] | [HOST2.EXAMPLE.COM] | +--------------------------------------+---------------------+

  • Nova API Pod Logs:

    • First grep the Nova API logs with compute service UUID and Note the Deletion Request ID.

$ kubectl logs <NOVA_API_POD_NAME> -n <REGION_NAMESPACE> | grep <COMPUTE_SERVICE_UUID> INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756

- Now grep the Nova API logs with Deletion Request ID and Note the Missing Resource Provider UUID.

$ kubectl logs <NOVA_API_POD_NAME> -n <REGION_NAMESPACE> | grep <DELETION_REQUEST_ID> Defaulted container "nova-osapi" out of: nova-osapi, socat18774, init (init) ERROR nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] Unexpected exception in API method: ValueError: No such provider [MISSING_RESOURCE_PROVIDER_UUID] INFO nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'ValueError'> INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_IDD] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756 /var/lib/openstack/lib/python3.10/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: unknown. To ensure interoperability, identifiers should be a valid uuid.

  • Based on above logs, as part of compute service deletion Nova is trying to delete the associated resource provider in Placement, but it's unable to locate it in the ProviderTree.

Resolution

  • Created a dummy Resource Provider with the same UUID that Nova is looking for during compute service deletion.

$ openstack resource provider create --uuid=<MISSING_RESOURCE_PROVIDER_UUID> temp_provider Expecting value: line 1 column 1 (char 0)

  • Verify if the dummy Resource Provider with the same UUID is created.

$ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+ | uuid | name | +--------------------------------------+---------------------+ | [RESOURCE_PROVIDER_UUID_1] | [HOST1.EXAMPLE.COM] | | [RESOURCE_PROVIDER_UUID_1] | [HOST2.EXAMPLE.COM] | | [MISSING_RESOURCE_PROVIDER_UUID1] | temp_provider | +--------------------------------------+---------------------+

  • Now, Delete the Compute service.

$ openstack compute service delete <COMPUTE_SERVICE_UUID>

Validation

  • Check the Complete Services list.

$ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+-------------------+---------+--------+ | ID | Binary | Host | Status | State | +---------------------------+----------------+-------------------+----------+-------+ | [COMPUTE_SERVICE_UUID_1] | nova-scheduler | nova-scheduler | enabled | up | | [COMPUTE_SERVICE_UUID_2] | nova-conductor | nova-conductor | enabled | up | +---------------------------+----------------+-------------------+----------+-------+

Additional Information

  • Please consult our documentation for the recommended procedure for offboarding a host from PCD.


Next to read:
Live Migration Failure Due to CPU Model Mismatch
null
On This Page
Unable to Delete Stale Compute ServiceProblemEnvironmentCauseDiagnosticsResolutionValidationAdditional Information