Delete Orphaned Allocations Using Nova-Manage

Issue

• When an instance is deleted/migrated/evacuated, sometimes it's corresponding allocation on source host is not deleted from placement and they remain stale.

Resolution

• Query placement API to list all the stale allocations. The below command will list allocations for instances that are either deleted or moved to other compute nodes. Run the below command from any one compute node.

xxxxxxxxxx

$ kubectl exec -it deploy/nova_api_osapi -n <NS> -- bash

$ nova-manage placement audit --verbose

• Verify if the instance exists or is on another compute node.

xxxxxxxxxx

$ openstack server list --all-projects --host <Source Host>

$ openstack server show -c OS-EXT-SRV-ATTR:host <instance-ID>

• Identify the resource provider ID

xxxxxxxxxx

$ openstack resource provider list

• After validating the allocation is stale and the instance is deleted or moved to other compute node, delete the orphaned allocation using the below command.

xxxxxxxxxx

$  kubectl exec -it <nova_api_osapi-pod_name> -n <NS> -- bash

$  nova-manage placement audit --verbose --delete <instance-ID>

• If there are multiple orphaned allocations, delete them altogether with the below command.

xxxxxxxxxx

$ nova-manage placement audit --verbose --delete

• After running above we can run a heal allocations for placement to ensure that all placement entries are consistent

xxxxxxxxxx

$ nova-manage placement heal_allocations

• To validate that there are no more orphaned allocations , please run below command # nova-manage placement audit --verbose

Root Cause

• When evacuating an instance if the compute service on the source host is stopped then the records in placement are not deleted automatically.
• As a result, ostackhost service cannot communicate with nova-conductor service running on the controller so details are not shared there and it results nova-compute remains in an impression that still holds the instance.