LTS3 Release Notes

Platform9 SMCP LTS3 - Patch #2

2024-03-04

Features/Enhancements

Added Workload clusters can now be created with Kubernetes versions 1.25, 1.26 (with higher patches for each minor kube version with updated components, addons) and 1.27.

Added Added support for k8s 1.27 workload clusters. Removed support for the deprecated k8s 1.24 (all workload clusters must have been upgraded to k8s 1.25 / 1.26 before upgrading to SMCP 5.9.2)

Added Introducing dual-stack networking with Multus support.

Added Upgraded Whereabouts to v0.6.3 to address the issue where new job pods for the ip-reconciler cronjob were unable to start (Fixes AIR-1268).

Added Verified the management plane upgrade from 5.9.0/5.9.1 to 5.9.2.

Added Implemented Support Bundle UX improvements.

Bug Fixes

Fixed Resolved AIR-1287: Nodelet service restart no longer resets any custom kubelet config applied. (Ticket #1458123)

Fixed Addressed AIR-1273: Automatic logrotate cronjob for Fluentbit now works correctly without state file lock issues. (Ticket #1456693)

Fixed Fixed AIR-1275: Resolved OOMKILL issue for pf9-nginx pod during DU installation by increasing memory limit to 200M. (Ticket #1457616, Ticket #1458222)

Fixed Resolved AIR-1289: Management plane password updating via CLI now correctly updates. (Ticket #1458115)

Fixed Addressed AIR-1272: The hosts section in coredns configmap no longer loses previous entries after airctl start.

Fixed Upgraded Luigi to v0.5.4, resolving issues (Ticket #1458596, Ticket #1457150)

Fixed Fixed AIR-1253: Resolved Luigi addon error state on 1.26 cluster due to cert manager timeout.

Known Issues

Known Issue When upgrading workload clusters from older kube versions shipped with 5.9.0/5.9.1, Grafana addon needs to be disabled and enabled again before it functions properly.

Known Issue Grafana has been downgraded to v7 to address issues with Grafana v10.2 in LTS3 5.9.1.

Known Issue AIR-1291: airctl upgrade-mgmt command is broken and fails to upgrade management cluster. This will be addressed in SMCP 5.10.1

SMCP LTS3 Kubernetes Components List

Platform9 SMCP LTS3 - Patch #1

2023-12-08

Release Highlights

Kubernetes 1.26 Support

Added support for Kubernetes 1.26 for workload clusters.

Security Fixes

Resolved all CRITICAL CVEs for 1.25 and 1.26 Kubernetes. This also includes upstream fixes to components for any CVEs rated HIGH. You can download the security scanning reports from the following links.

1.25-kubebench-report | 1.26-kubebench-report

Features/Enhancements

Added Various improvements to support bundle generation.

Added Dual stack support for the management plane. The workload clusters still work only on a single stack.

Added Support for running the management plane and workload clusters in IPv6.

Bug Fixes

Fixed Deprecated the corednsHosts field in bootstrap config file. Before the management cluster is created, add the following to the bootstrap config file:

xxxxxxxxxx

echo "34.35.69.42 s3-us-west-1.amazonaws.com" >> /path/to/custom/hosts

# Then, Update the bootstrap config file with:

dns:

  hostsFile: /path/to/custom/hosts

Alternatively, you can run your own DNS server that can resolve this IP.

Fixed Fixed an issue with the logrotation script that was causing the kubernetes logs to grow uncontrollably.

Fixed Fixed an issue with the migration of the management plane from LTS2 Patch 2 to LTS3.

Fixed Fixed an issue that would cause airctl host-status to report all host IPs, not just the primary one.

Fixed Fixed an issue where a failed airctl configure-hosts command was leaving behind artifacts in the host’s /tmp directory.

Fixed Raised the memory limits to 100Mi for the clarity and serenity pods to account for higher resource usage.

Known Issues

Known Issue Kubelet logs on a worker frequently report not being able to find /etc/pf9/kube.d/pod-manifests. This is a benign warning. It ignores the directory if it is not present.

Known Issue Keystone user passwords do not accept special characters

Known Issue During upgrade from LTS2 to LTS3, 1.24 clusters might have the kube-state-metrics pod go into an image pull backoff. This will not break the upgrade, and will get resolved once upgraded to 1.25 or higher.

Known Issue Upgrades could fail because of missing images due to an over zealous containerd garbage collector cleaning up the images that are needed during upgrade. Please take the below steps to recover from this failure:

1. Remove the checksum file so that nodelet would reload the images on restart.

xxxxxxxxxx

sudo rm /var/opt/pf9/images/checksum/sha256sums.txt

2. Restart the nodelet cluster so that the kubedu images are loaded again

xxxxxxxxxx

sudo /opt/pf9/nodelet/nodeletd phases restart

3. Try the DU upgrade again using

xxxxxxxxxx

airctl upgrade

Platform9 SMCP LTS3 (GA)

2023-09-12

Release Highlights

Online mode

Added support for running the management plane in connected mode. Prior releases were designed around running the management plane in an air-gapped environment and required users to download all artifacts prior to deployment. With LTS3, the management plane can run connected to the internet and the artifacts are downloaded just-in-time, significantly reducing the initial startup time.

Management Plane, Management Cluster Upgrade

The management cluster at the heart of the containerized management plane is now based on k8s v1.25 (from v1.21 in LTS2).

Ability to upgrade LTS2 management plane to LTS3 with backup-restore

Environments running on LTS2 can migrate to LTS3. The Platform9 Management Plane will require a downtime as the migration is done, but the user's workload clusters will continue to run.

Support for Rocky Linux

Rocky Linux 9.2 is now supported in addition to CentOs 7.9 and RHEL 8.5, 8.6

Features/ Updates

Added Support for Rocky 9.2 Linux

Added API to create etcd, kubelet to use specific IP address/network interface (on workload clusters)

Added (1397740) Add the ability to perform Kubelet customization prior to cluster creation (handled using the dynamic kubelet config feature in k8s 1.24+)

Bug fixes

Fixed (1454832, 1454841) Ip-reconciler pod stuck in Running state - No new ip-reconciler job getting created in every 5 mins

Fixed (1454058) calico-node pods failing in v-3.18: felix is not ready: readiness probe reporting 503 (fixed with calico upgrade to v3.24.2)

Fixed (1453914) Updating API audit parameters using qbert API leads to duplicate entries.

Fixed (1452055,1452533) Liveness and Readiness probe failures for percona db haproxy pod

Fixed (1396273) Sign pf9 component packages (pf9-hostagent and pf9-kube were missing signatures)

Fixed (1393131) In case of Network fluctuations avoid assigning VIP to all Masters by Keepalived

Fixed (1385822) Displaying host upgrade status with airctl

Security

Enhanced Improved security by updating upstream images to the latest versions which had vulnerability fixes implemented.

Known Issues

Known Issue (1454408) Onboarding node with pf9ctl fails with error "Failed to authenticate with the keystone server" when user has special character in the password

Known Issue (1398205) some sriov resources are not reflecting in node description after node reboot (customer is responsible for restarting the sriov plugin pod to pick up the new VFs, and this may cause connectivity issues on existing pods using SRIOV)

Known Issue (1450125) Pod to pod communication over calico not working in ipv6 setup if node ips are in different subnet

Known Issue (1454599) vlan not getting attached to the VF on relocated pod due to worker node reboot

Known Issue SMCP management cluster deletion process with 'delete-mgmt' doesn't delete all the files/directories of the previous installation

Known Issue Support for IPV6 has not been verified

Known Issue airctl status when run as root user shows json parsing error

Known Issue airctl renew-certs does not renew certs when all certs including CA are external

Known Issue airctl configuration should not accept same ip for multiple masters