Services Fail After Accidental Deletion of Regional MySQL Databases

Problem

In certain cases, the MySQL database for a Self-hosted Private Cloud Director region may be accidentally deleted or corrupted. This can cause critical platform components to fail, resulting in an unusable environment and loss of region-specific data.

Environment

  • Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher

  • Component - MySQL

Cause

Accidental deletion or corruption of the MySQL database for a specific region(s).

Diagnostics

When a regional MySQL database is deleted or corrupted, symptoms may include:

  • Components failing to start or repeatedly crashing.

  • Errors related to missing tables or database connections in component/pod logs.

  • The region appearing as unreachable or inactive in the control plane UI.

To confirm database loss login to MySQL from Percona pod:

$ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bash bash-4.4$ mysql -u root -p<PASSWORD> mysql>show databases; [!DATABASE MISSING!]

If the region databases are missing from the list, restoration from the latest backup is required.

Resolution

As part of the Self-hosted Private Cloud Director backup process, MySQL data — including users and permissions — is backed up automatically for each region [Backup guide]. To restore the database without performing a full region restore, follow the steps below:

IMPORTANT NOTE

Before proceeding with the MySQL restore process, MySQL root password needs to be retrieved from the Kubernetes secret in its corresponding region namespace using

kubectl get secret -n <REGION_NAME> mysql -o jsonpath='{.data.pass}' | base64 -d

  1. Extract the Backup

Locate the backup archive obtained from the automated or manual backup process and extract it:

$ tar -xzvf backups/backup_20251113_181414.tar.gz backups/state_backup.yaml backups/kplane_values_backup.yaml backups/consul.snap backups/mysql_dump_Infra.sql backups/mysql_dump_LON.sql backups/ovn-north-backup-LON-ovn-ovsdb-nb-0.tar.gz backups/ovn-north-backup-LON-ovn-ovsdb-nb-1.tar.gz backups/ovn-north-backup-LON-ovn-ovsdb-nb-2.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-0.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-1.tar.gz backups/ovn-south-backup-LON-ovn-ovsdb-sb-2.tar.gz backups/rabbitmq-backup-Infra.tar.gz backups/rabbitmq-backup-LON.tar.gz

Within the extracted backup, locate the MySQL dump file for the affected region mysql_dump_<REGION_NAME>.sql

For example, for the Infra region backups/mysql_dump_Infra.sql

  1. Scale Down Percona Replicas

To safely restore the database, first scale down the Percona PXC replicas for the region:

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=0 statefulset.apps/percona-db-pxc-db-pxc scaled

  1. Delete Existing Percona PVCs

Delete the existing PersistentVolumeClaims (PVCs) associated with the Percona MySQL pods to remove the corrupted data:

$ kubectl get pvc -n <REGION_NAME> | grep percona-db-pxc-db-pxc datadir-percona-db-pxc-db-pxc-0 Bound [PVC_UUID1] 96Gi RWO hostpath-csi <unset> 3d6h datadir-percona-db-pxc-db-pxc-1 Bound [PVC_UUID2] 96Gi RWO hostpath-csi <unset> 3d6h datadir-percona-db-pxc-db-pxc-2 Bound [PVC_UUID3] 96Gi RWO hostpath-csi <unset> 3d6h $ kubectl delete pvc datadir-percona-db-pxc-db-pxc-0 datadir-percona-db-pxc-db-pxc-1 datadir-percona-db-pxc-db-pxc-2 -n <REGION_NAME> persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-0" deleted persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-1" deleted persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-2" deleted

  1. Scale Up Percona Replicas

Scale the replicas back up to recreate the database with new, empty volumes:

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=3 statefulset.apps/percona-db-pxc-db-pxc scaled $ kubectl get pods -n <REGION_NAME> | grep percona-db-pxc-db-pxc percona-db-pxc-db-pxc-0 3/3 Running 0 3m26s percona-db-pxc-db-pxc-1 3/3 Running 0 2m25s percona-db-pxc-db-pxc-2 3/3 Running 0 83s

  1. Restore the Database from Backup

Once the Percona pods are up and running, copy the regional MySQL dump into percona-db-pxc-db-pxc-0 pod:

$ kubectl cp backups/mysql_dump_Infra.sql percona-db-pxc-db-pxc-0:/tmp -n <REGION_NAME> -c pxc $ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bash bash-4.4$ ls /tmp/mysql_dump_Infra.sql /tmp/mysql_dump_Infra.sql ## No platform9 service bash-4.4$ mysql -u root -p<PASSWORD> mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec)

Then, restore the dump using the MySQL client:

bash-4.4$ mysql -u root -p<PASSWORD> < <PATH_TO_MYSQL_BACKUP.sql> && mysql -u root -p<PASSWORD> -e 'flush privileges' mysql: [Warning] Using a password on the command line interface can be insecure. mysql: [Warning] Using a password on the command line interface can be insecure.

Validation

  • Log in to the database from percona pod and confirm that the expected databases and tables are present:

mysql> show databases; +--------------------+ | Database | +--------------------+ | alertmanager | | hagrid | | information_schema | | keystone | | mysql | | performance_schema | | preference_store | | resmgr | | sys | +--------------------+ 9 rows in set (0.00 sec)

  • Verify that services are running and the region is healthy:

$ /opt/pf9/airctl/airctl status --region <REGION_NAME> --config /opt/pf9/airctl/conf/airctl-config.yaml ------------- deployment details --------------- fqdn: [FQDN] region: [REGION_NAME] deployment status: ready region health: ✅ Ready version: PCD 2025.10-3111 -------- region service status ---------- desired services: 30 ready services: 30

  • Access the management plane UI and confirm normal operations , API responses, and workflows should function as expected.


Next to read:
Host Decommission: Incomplete Platform9 Cleanup
null
On This Page
Services Fail After Accidental Deletion of Regional MySQL DatabasesProblemEnvironmentCauseDiagnosticsResolutionValidation