Services Fail After Accidental Deletion of Regional MySQL Databases

Problem

In certain cases, the MySQL database for a Self-hosted Private Cloud Director region may be accidentally deleted or corrupted. This can cause critical platform components to fail, resulting in an unusable environment and loss of region-specific data.

Environment

  • Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher

  • Component - MySQL

Cause

Accidental deletion or corruption of the MySQL database for a specific region(s).

Diagnostics

When a regional MySQL database is deleted or corrupted, symptoms may include:

  • Components failing to start or repeatedly crashing.

  • Errors related to missing tables or database connections in component/pod logs.

  • The region appearing as unreachable or inactive in the control plane UI.

To confirm database loss login to MySQL from Percona pod:

$ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bash

bash-4.4$ mysql -u root -p<PASSWORD>

mysql>show databases;
[!DATABASE MISSING!]

If the region databases are missing from the list, restoration from the latest backup is required.

Resolution

As part of the Self-hosted Private Cloud Director backup process, MySQL data — including users and permissions — is backed up automatically for each region [Backup guide]. To restore the database without performing a full region restore, follow the steps below:

IMPORTANT NOTE

Before proceeding with the MySQL restore process, MySQL root password needs to be retrieved from the Kubernetes secret in its corresponding region namespace using

kubectl get secret -n <REGION_NAME> mysql -o jsonpath='{.data.pass}' | base64 -d

  1. Extract the Backup

Locate the backup archive obtained from the automated or manual backup process and extract it:

$ tar -xzvf backups/backup_20251113_181414.tar.gz
backups/state_backup.yaml
backups/kplane_values_backup.yaml
backups/consul.snap
backups/mysql_dump_Infra.sql
backups/mysql_dump_LON.sql
backups/ovn-north-backup-LON-ovn-ovsdb-nb-0.tar.gz
backups/ovn-north-backup-LON-ovn-ovsdb-nb-1.tar.gz
backups/ovn-north-backup-LON-ovn-ovsdb-nb-2.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-0.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-1.tar.gz
backups/ovn-south-backup-LON-ovn-ovsdb-sb-2.tar.gz
backups/rabbitmq-backup-Infra.tar.gz
backups/rabbitmq-backup-LON.tar.gz

Within the extracted backup, locate the MySQL dump file for the affected region mysql_dump_<REGION_NAME>.sql

For example, for the Infra region backups/mysql_dump_Infra.sql

  1. Scale Down Percona Replicas

To safely restore the database, first scale down the Percona PXC replicas for the region:

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=0
statefulset.apps/percona-db-pxc-db-pxc scaled

  1. Delete Existing Percona PVCs

Delete the existing PersistentVolumeClaims (PVCs) associated with the Percona MySQL pods to remove the corrupted data:

$ kubectl get pvc -n <REGION_NAME> | grep percona-db-pxc-db-pxc
datadir-percona-db-pxc-db-pxc-0 Bound [PVC_UUID1] 96Gi RWO hostpath-csi <unset> 3d6h
datadir-percona-db-pxc-db-pxc-1 Bound [PVC_UUID2] 96Gi RWO hostpath-csi <unset> 3d6h
datadir-percona-db-pxc-db-pxc-2 Bound [PVC_UUID3] 96Gi RWO hostpath-csi <unset> 3d6h

$ kubectl delete pvc datadir-percona-db-pxc-db-pxc-0 datadir-percona-db-pxc-db-pxc-1 datadir-percona-db-pxc-db-pxc-2 -n <REGION_NAME>
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-0" deleted
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-1" deleted
persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-2" deleted

  1. Scale Up Percona Replicas

Scale the replicas back up to recreate the database with new, empty volumes:

$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=3
statefulset.apps/percona-db-pxc-db-pxc scaled

$ kubectl get pods -n <REGION_NAME> | grep percona-db-pxc-db-pxc
percona-db-pxc-db-pxc-0 3/3 Running 0 3m26s
percona-db-pxc-db-pxc-1 3/3 Running 0 2m25s
percona-db-pxc-db-pxc-2 3/3 Running 0 83s

  1. Restore the Database from Backup

Once the Percona pods are up and running, copy the regional MySQL dump into percona-db-pxc-db-pxc-0 pod:

$ kubectl cp backups/mysql_dump_Infra.sql percona-db-pxc-db-pxc-0:/tmp -n <REGION_NAME> -c pxc

$ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bash
bash-4.4$ ls /tmp/mysql_dump_Infra.sql
/tmp/mysql_dump_Infra.sql

## No platform9 service
bash-4.4$ mysql -u root -p<PASSWORD>

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)

Then, restore the dump using the MySQL client:

bash-4.4$ mysql -u root -p<PASSWORD> < <PATH_TO_MYSQL_BACKUP.sql> && mysql -u root -p<PASSWORD> -e 'flush privileges'
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.

Validation

  • Log in to the database from percona pod and confirm that the expected databases and tables are present:

mysql> show databases;
+--------------------+
| Database |
+--------------------+
| alertmanager |
| hagrid |
| information_schema |
| keystone |
| mysql |
| performance_schema |
| preference_store |
| resmgr |
| sys |
+--------------------+
9 rows in set (0.00 sec)

  • Verify that services are running and the region is healthy:

$ /opt/pf9/airctl/airctl status --region <REGION_NAME> --config /opt/pf9/airctl/conf/airctl-config.yaml
------------- deployment details ---------------
fqdn: [FQDN]
region: [REGION_NAME]
deployment status: ready
region health: ✅ Ready
version: PCD 2025.10-3111
-------- region service status ----------
desired services: 30
ready services: 30

  • Access the management plane UI and confirm normal operations , API responses, and workflows should function as expected.


On This Page
Services Fail After Accidental Deletion of Regional MySQL DatabasesProblemEnvironmentCauseDiagnosticsResolutionValidation