Upgrade Guide: PEC 5.6 to SMCP 5.9.1

Prerequisites

Before beginning the upgrade process, ensure the following requirements are met:

Ensure that the Minimum Requirements are met.
Minimum Kubernetes version for Workload Cluster should be v1.22.

Upgrade Procedure

1. Backup Current System

Follow the Backup Guide to back up the 5.6 management plane. Before proceeding, verify that the backup was created successfully.

2. Take etcd Snapshot

Before making any changes to the system, it's crucial to take a snapshot of the etcd cluster. This provides an additional recovery point if needed.

a. Check the etcd cluster status and identify the leader node:

sudo /opt/pf9/pf9-kube/bin/etcdctl endpoint status --cluster -w=table --endpoints=http://127.0.0.1:2379 --cacert="/etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt" --cert="/etc/pf9/kube.d/certs/etcdctl/etcd/request.crt" --key="/etc/pf9/kube.d/certs/etcdctl/etcd/request.key"

Make a note of which node is the leader.

b. Take the etcd cluster backup:

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot save <path/backupfile.db> --cert /etc/pf9/kube.d/certs/etcdctl/etcd/request.crt --key /etc/pf9/kube.d/certs/etcdctl/etcd/request.key --cacert /etc/pf9/kube.d/certs/etcdctl/etcd/ca.crt --endpoints=https://<master_IP>:4001

Replace <path/backupfile.db> with the desired path and filename for your backup and <master_IP> with the IP address of the master node.

c. Validate the etcd backup:

sudo /opt/pf9/pf9-kube/bin/etcdctl snapshot status --write-out=table <path/backupfile.db>

Ensure that the snapshot status shows as valid before proceeding.

3. Destroy Old Cluster

Remove the existing 1.22 based nodelet cluster:

airctl stop # Stops all running services managed by airctl airctl unconfigure-du --force # Removes DU configuration and forcefully cleans up resources airctl advanced-ddu delete-mgmt # Deletes the management plane components

4. Clean Up Configuration

On all master nodes, clean up the certificate directory:

rm -rf /etc/nodelet/airctl-mgmt/certs/* # This command removes all files in the specified directory

Verify the directory is empty before proceeding.

5. Download and Install New Artifacts

To obtain and install the SMCP 5.9.1 artifacts:

a. Download the download script:

wget https://raw.githubusercontent.com/platform9/support-locker/master/edge-stack/download.sh

b. Make the script executable:

chmod u+x download.sh

c. Run the download script:

./download.sh <PF9-SHARED-SECRET> v-5.9.1-<airctl_version>

Replace <PF9-SHARED-SECRET> with your Platform9 shared secret.

d. After downloading the artifacts, verify their integrity using the following command:

curl --user-agent <secret> https://pf9-airctl.s3-accelerate.amazonaws.com/<version>/md5sum.txt | md5sum -c -

Replace <secret> with the same one used to download the airctl build from S3 and <version> with the SMCP version you're installing (e.g., v5.9.1).

e. Run the install.sh script to extract and place the artifacts under /opt/pf9/airctl directory.

bash install.sh v-5.9.1-<airctl_version>

Refer to the SMCP 5.9 release notes for the exact airctl version used by the install.sh script.

For example, when upgrading to SMCP v5.9.1 :

bash install.sh v-5.9.1-3097398

6. Configure New Management Cluster

Update configuration files before deployment:

Edit airctl-config.yaml:
1. Update hostAgentRepo to use SMCP 5.9.1 images.
2. Update dockerRepo to use SMCP 5.9.1 images.
Edit nodelet-bootstrap.yaml:
1. Update systemImages to use SMCP 5.9.1 image files
2. Set userImages to point to PEC 5.6.7 image files
3. Add isAirgapped: true for offline installations

Example airctl-config.yaml and nodelet-bootstrap.yaml snippet:

hostAgentRepo: /opt/pf9/airctl/hostagent-8-v-5.9.1-<build_number>.tar.gz dockerRepo: /opt/pf9/airctl/docker-8-v-5.9.1-<build_number>.tar.gz

isAirgapped: true systemImages: - /opt/pf9/airctl/imgs/kubedu-imgs-v-5.9.1-<build_number>.tar.gz - /opt/pf9/airctl/imgs/nodelet-imgs-v-5.9.1-<build_number>.tar.gz userImages: - /opt/pf9/airctl/imgs/kubedu-imgs-v-5.6.7-<build_number>.tar.gz

7. Deploy New Management Cluster

Deploy the new 1.25 based nodelet management cluster:

airctl advanced-ddu create-mgmt --verbose # This command creates and deploys the new management plane components

Initialize the management plane:

sudo airctl init

Monitor the deployment progress and check for any errors.

8. Restore from Backup

Restore the PEC 5.6 backup by following the restore guide for the 5.9.1 management plane on top of the 1.25 nodelet cluster.

Consul Issue:

After restoring, you might see errors in the Consul pod saying ACL not found. These errors are harmless and can be ignored.

Therefore, SMCP currently skips the Consul upgrade in the upgrade step and installs the upgraded Consul during the restore process itself.

9. Verify DU Status

Check that the DU is in a ready state:

airctl status # This command displays the current status of the DU

Wait until all components show as ready before proceeding.

10. Upgrade DU Components

To upgrade the DU components to5.9.1, follow the upgrade guide.

NOTE

To prevent potential ImagePullBackOff errors, refer to additional-information

After upgrade, verify DU status again:

airctl status

11. Verify Upgrade in GUI

12. Push New Kubernetes Images

Push images for Kubernetes 1.24, 1.25, and 1.26 to your private registry (if applicable):

sudo airctl advanced-du push-images --img-tar [tar file name]

To push all new images at once:

ls -1 /opt/pf9/airctl/imgs/k8s* | xargs -I{} sh -c "sudo /opt/pf9/airctl/airctl --config /opt/pf9/airctl/conf/airctl-config.yaml advanced-du push-images --img-tar {}"

NOTE

If the privateRegistryType value is custom, the images must be manually pushed to the custom registry.

13. Upgrade Host Components

Upgrade the host-side components:

airctl configure-hosts --reset-pf9-managed-docker-conf --skip-docker-install --skip-docker-img-import --verbose # This command configures hosts with new settings airctl hosts-status # Check the status of hosts before proceeding airctl upgrade-hosts --verbose # This command upgrades the host components

14. Verify Status and Version

After upgrading, verify management plane services are ready. Run the following command to verify.

airctl status # Check the status of management plane airctl version # The output should be: v-5.9.1-3097398 kubectl get nodes -o wide # Ensure all the nodes are in Ready state

At this point all the components have been upgraded to SMCP 5.9.1 (LTS3).

Now, proceed with the actual workload cluster upgrades either using the GUI or API.

Last updated on

Was this page helpful?