Title
Create new category
Edit page index title
Edit category
Edit link
Services Fail After Accidental Deletion of Regional MySQL Databases
Problem
In certain cases, the MySQL database for a Self-hosted Private Cloud Director region may be accidentally deleted or corrupted. This can cause critical platform components to fail, resulting in an unusable environment and loss of region-specific data.
Environment
- Self-Hosted Private Cloud Director Virtualization - v2025.10 and Higher
- Component - MySQL
Cause
Accidental deletion or corruption of the MySQL database for a specific region(s).
Diagnostics
When a regional MySQL database is deleted or corrupted, symptoms may include:
- Components failing to start or repeatedly crashing.
- Errors related to missing tables or database connections in component/pod logs.
- The region appearing as unreachable or inactive in the control plane UI.
To confirm database loss login to MySQL from Percona pod:
$ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bash bash-4.4$ mysql -u root -p<PASSWORD>mysql>show databases;[!DATABASE MISSING!]If the region databases are missing from the list, restoration from the latest backup is required.
Resolution
As part of the Self-hosted Private Cloud Director backup process, MySQL data — including users and permissions — is backed up automatically for each region [Backup guide]. To restore the database without performing a full region restore, follow the steps below:
Before proceeding with the MySQL restore process, MySQL root password needs to be retrieved from the Kubernetes secret in its corresponding region namespace using
kubectl get secret -n <REGION_NAME> mysql -o jsonpath='{.data.pass}' | base64 -d
- Extract the Backup
Locate the backup archive obtained from the automated or manual backup process and extract it:
$ tar -xzvf backups/backup_20251113_181414.tar.gz backups/state_backup.yamlbackups/kplane_values_backup.yamlbackups/consul.snapbackups/mysql_dump_Infra.sqlbackups/mysql_dump_LON.sqlbackups/ovn-north-backup-LON-ovn-ovsdb-nb-0.tar.gzbackups/ovn-north-backup-LON-ovn-ovsdb-nb-1.tar.gzbackups/ovn-north-backup-LON-ovn-ovsdb-nb-2.tar.gzbackups/ovn-south-backup-LON-ovn-ovsdb-sb-0.tar.gzbackups/ovn-south-backup-LON-ovn-ovsdb-sb-1.tar.gzbackups/ovn-south-backup-LON-ovn-ovsdb-sb-2.tar.gzbackups/rabbitmq-backup-Infra.tar.gzbackups/rabbitmq-backup-LON.tar.gzWithin the extracted backup, locate the MySQL dump file for the affected region mysql_dump_<REGION_NAME>.sql
For example, for the Infra region backups/mysql_dump_Infra.sql
- Scale Down Percona Replicas
To safely restore the database, first scale down the Percona PXC replicas for the region:
$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=0statefulset.apps/percona-db-pxc-db-pxc scaled- Delete Existing Percona PVCs
Delete the existing PersistentVolumeClaims (PVCs) associated with the Percona MySQL pods to remove the corrupted data:
$ kubectl get pvc -n <REGION_NAME> | grep percona-db-pxc-db-pxcdatadir-percona-db-pxc-db-pxc-0 Bound [PVC_UUID1] 96Gi RWO hostpath-csi <unset> 3d6hdatadir-percona-db-pxc-db-pxc-1 Bound [PVC_UUID2] 96Gi RWO hostpath-csi <unset> 3d6hdatadir-percona-db-pxc-db-pxc-2 Bound [PVC_UUID3] 96Gi RWO hostpath-csi <unset> 3d6h$ kubectl delete pvc datadir-percona-db-pxc-db-pxc-0 datadir-percona-db-pxc-db-pxc-1 datadir-percona-db-pxc-db-pxc-2 -n <REGION_NAME>persistentvolumeclaim "datadir-percona-db-pxc-db-pxc-0" deletedpersistentvolumeclaim "datadir-percona-db-pxc-db-pxc-1" deletedpersistentvolumeclaim "datadir-percona-db-pxc-db-pxc-2" deleted- Scale Up Percona Replicas
Scale the replicas back up to recreate the database with new, empty volumes:
$ kubectl scale sts percona-db-pxc-db-pxc -n <REGION_NAME> --replicas=3statefulset.apps/percona-db-pxc-db-pxc scaled$ kubectl get pods -n <REGION_NAME> | grep percona-db-pxc-db-pxcpercona-db-pxc-db-pxc-0 3/3 Running 0 3m26spercona-db-pxc-db-pxc-1 3/3 Running 0 2m25spercona-db-pxc-db-pxc-2 3/3 Running 0 83s- Restore the Database from Backup
Once the Percona pods are up and running, copy the regional MySQL dump into percona-db-pxc-db-pxc-0 pod:
$ kubectl cp backups/mysql_dump_Infra.sql percona-db-pxc-db-pxc-0:/tmp -n <REGION_NAME> -c pxc$ kubectl exec -it percona-db-pxc-db-pxc-0 -n <REGION_NAME> -c pxc -- bashbash-4.4$ ls /tmp/mysql_dump_Infra.sql /tmp/mysql_dump_Infra.sql## No platform9 servicebash-4.4$ mysql -u root -p<PASSWORD>mysql> show databases;+--------------------+| Database |+--------------------+| information_schema || mysql || performance_schema || sys |+--------------------+4 rows in set (0.00 sec)Then, restore the dump using the MySQL client:
bash-4.4$ mysql -u root -p<PASSWORD> < <PATH_TO_MYSQL_BACKUP.sql> && mysql -u root -p<PASSWORD> -e 'flush privileges'mysql: [Warning] Using a password on the command line interface can be insecure.mysql: [Warning] Using a password on the command line interface can be insecure.Validation
- Log in to the database from percona pod and confirm that the expected databases and tables are present:
mysql> show databases;+--------------------+| Database |+--------------------+| alertmanager || hagrid || information_schema || keystone || mysql || performance_schema || preference_store || resmgr || sys |+--------------------+9 rows in set (0.00 sec)- Verify that services are running and the region is healthy:
$ /opt/pf9/airctl/airctl status --region <REGION_NAME> --config /opt/pf9/airctl/conf/airctl-config.yaml ------------- deployment details ---------------fqdn: [FQDN]region: [REGION_NAME]deployment status: readyregion health: ✅ Readyversion: PCD 2025.10-3111-------- region service status ----------desired services: 30ready services: 30- Access the management plane UI and confirm normal operations , API responses, and workflows should function as expected.