Unable to Delete Stale Compute Service

Problem

Unable to delete stale compute service with the below error.

openstack command
    
​x
 
$ openstack compute service delete  [COMPUTE_SERVICE_UUID] Failed to delete compute service with ID '[COMPUTE_SERVICE_UUID]': HttpException: 500: Server Error for url: https://[example.platform9.com]/nova/v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID], Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.<class 'ValueError'>1 of 1 compute services failed to delete.
Copy

Environment

Private Cloud Director Virtualization - v2025.4 and Higher
Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher
Nova

Cause

Forcefully removing a host from PCD, resulting in incomplete cleanup of resources on the management plane.

Diagnostics

Compute Service List:

Openstack Command
    
xxxxxxxxxx
 
$ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+---------------------+---------+--------+| ID                        | Binary         | Host                | Status   | State |+---------------------------+----------------+---------------------+----------+-------+| [COMPUTE_SERVICE_UUID_1]  | nova-scheduler | nova-scheduler      | enabled  | up    || [COMPUTE_SERVICE_UUID_2]  | nova-conductor | nova-conductor      | enabled  | up    || [COMPUTE_SERVICE_UUID_3]  | nova-compute   | [HOST1.EXAMPLE.COM] | enabled  | up    |+---------------------------+----------------+---------------------+----------+-------+
Copy

Resource Provider List:

Openstack Command
    
xxxxxxxxxx
 
$ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+| uuid                                 | name                |+--------------------------------------+---------------------+| [RESOURCE_PROVIDER_UUID_1]           | [HOST1.EXAMPLE.COM] || [RESOURCE_PROVIDER_UUID_2]           | [HOST2.EXAMPLE.COM] |+--------------------------------------+---------------------+
Copy

Nova API Pod Logs:
- First grep the Nova API logs with compute service UUID and Note the Deletion Request ID.

Nova API logs
    
 
$ kubectl logs <NOVA_API_POD_NAME> -n <REGION_NAMESPACE> | grep <COMPUTE_SERVICE_UUID>  INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756
Copy

- Now grep the Nova API logs with Deletion Request ID and Note the Missing Resource Provider UUID.

Nova API Logs
    
 
$ kubectl logs <NOVA_API_POD_NAME> -n <REGION_NAMESPACE> | grep <DELETION_REQUEST_ID>  Defaulted container "nova-osapi" out of: nova-osapi, socat18774, init (init)ERROR nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] Unexpected exception in API method: ValueError: No such provider [MISSING_RESOURCE_PROVIDER_UUID]INFO nova.api.openstack.wsgi [None [DELETION_REQUEST_ID] [OTHER_ID_1] [OTHER_ID_2] - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.<class 'ValueError'>INFO nova.osapi_compute.wsgi.server [None [DELETION_REQUEST_IDD] [OTHER_ID_1] [OTHER_ID_2] - - default default] [IP_ADDRESS_1],[IP_ADDRESS_2] "DELETE /v2.1/[PROJECT_ID]/os-services/[COMPUTE_SERVICE_UUID] HTTP/1.1" status: 500 len: 595 time: 0.1055756/var/lib/openstack/lib/python3.10/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: unknown. To ensure interoperability, identifiers should be a valid uuid.
Copy

Based on above logs, as part of compute service deletion Nova is trying to delete the associated resource provider in Placement, but it's unable to locate it in the ProviderTree.

Resolution

Created a dummy Resource Provider with the same UUID that Nova is looking for during compute service deletion.

Openstack Command
    
xxxxxxxxxx
 
$ openstack resource provider create --uuid=<MISSING_RESOURCE_PROVIDER_UUID>  temp_provider Expecting value: line 1 column 1 (char 0)
Copy

Verify if the dummy Resource Provider with the same UUID is created.

Openstack Command
    
xxxxxxxxxx
 
$ openstack resource provider list -c uuid -c name +--------------------------------------+---------------------+| uuid                                 | name                |+--------------------------------------+---------------------+| [RESOURCE_PROVIDER_UUID_1]           | [HOST1.EXAMPLE.COM] || [RESOURCE_PROVIDER_UUID_1]           | [HOST2.EXAMPLE.COM] || [MISSING_RESOURCE_PROVIDER_UUID1]    | temp_provider       |+--------------------------------------+---------------------+
Copy

Now, Delete the Compute service.

Openstack Command
    
xxxxxxxxxx
 
$ openstack compute service delete <COMPUTE_SERVICE_UUID>
Copy

Validation

Check the Complete Services list.

Openstack Command
    
xxxxxxxxxx
 
$ openstack compute service list -c ID -c Binary -c Host -c Status -c State +---------------------------+----------------+-------------------+---------+--------+| ID                        | Binary         | Host              | Status   | State |+---------------------------+----------------+-------------------+----------+-------+| [COMPUTE_SERVICE_UUID_1]  | nova-scheduler | nova-scheduler    | enabled  | up    || [COMPUTE_SERVICE_UUID_2]  | nova-conductor | nova-conductor    | enabled  | up    |+---------------------------+----------------+-------------------+----------+-------+
Copy

Additional Information

Please consult our documentation for the recommended procedure for offboarding a host from PCD.

Last updated on

Was this page helpful?