Scaling Down the Management Cluster

In order to scale down the management cluster, a manual PV migration steps is required.

If the nodes to be scaled down host SeaweedFS or Consul deployments, the PVs attached to them must be migrated to other nodes in the cluster before scaling down.

Prerequisites

Identify which pods are on nodes being removed
Identify target nodes for migration
Ensure SSH/rsync access between nodes

Migration Process (Per Pod)

Get current state

# Find pod's current node

$ kubectl get pod <pod-name> -o wide

# Get PV name from PVC

$ kubectl get pvc <pvc-name> -o jsonpath='{.spec.volumeName}'

# Get current node IP from PV

$ kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}'

Prepare new node directory :

ssh ubuntu@<new-node-ip>

sudo mkdir -p /opt/pf9/volumes/csi/<pvc-id>

sudo chown root:ubuntu /opt/pf9/volumes/csi/<pvc-id>

sudo chmod 2775 /opt/pf9/volumes/csi/<pvc-id>

exit

Rsync data from old to new node :

# From old node

ssh ubuntu@<old-node-ip>

sudo rsync -avzAX --progress \

--rsync-path="sudo rsync" \

/opt/pf9/volumes/csi/<pvc-id>/ \

ubuntu@<new-node-ip>:/opt/pf9/volumes/csi/<pvc-id>/

Save PV configuration :

kubectl get pv <pv-name> -o yaml > pv-backup.yaml

Edit PV YAML

vim pv-backup.yaml

Changes needed:

Update nodeAffinity IP: old-node-ip → new-node-ip
Remove from metadata: creationTimestamp, resourceVersion, uid
Remove entire status: section

Delete PV and PVC :

kubectl apply -f pv-backup.yaml

Recreate PV only :

kubectl apply -f pv-backup.yaml

Delete pod - StatefulSet will recreate PVC and bind :

kubectl delete pod <pod-name>

Verify :

# Verify PV-PVC bind

kubectl get pv,pvc | grep <pvc-name>

# Verify pod is on new node

kubectl get pod <pod-name> -o wide

Last updated on

Was this page helpful?