One-click OpenShift Upgrade — It’s Reality

7 min readAug 28, 2021

Technology is always changing, which creates opportunities and challenges for businesses. There are lot of importance of staying currents on your system and application upgrades. Upgrades are critical for business performance. It provides many benefits like, greater efficiency, better security, enhancements, compatibility etc. While this is exciting, it also presents an array of unprecedented challenges. With technology being a major factor in the success or failure of business, it is critical that you carefully and strategically consider system and application upgrades. Just like any other technology, there is no difference in importance on upgrade, when you are dealing with Cloud Pak for Data and OpenShift.

This post will go through the OpenShift 4 upgrade process. You will see how simple the upgrade process. OpenShift 4 brings supreme automation to cluster installation, cluster scaling, maintenance, and updates. The one-click OpenShift upgrade is reality today.

Post will discuss about the over-the-air (OTA) upgrade process here. The over-the-air upgrade can be performed on an internet connected cluster. Red Hat provides this kind of updates through an OpenShift Container Platform update service as a hosted service located behind public APIs. If you are on a restricted network where cluster cannot access the public APIs, then it is all together a different story and you need to install the OpenShift Update Service locally.

High Level Architecture

The OpenShift Update Service (OSUS) provides over-the-air (OTA) updates to OpenShift Container Platform, including CoreOS. It provides a diagram, that contains the vertices of component Operators and the edges that connect them. The edges in the graph show which versions you can safely update. The vertices are update payloads that specify the intended state of the managed cluster components.

The Cluster Version Operator (CVO) in your cluster checks with the OpenShift Update Service to see the valid updates and update paths based on current component versions and information in the graph. When you request an update, the CVO uses the release image for that update to upgrade your cluster. The release artifacts are hosted in Quay as container images. New versions of all cluster Operators are downloaded, and their signatures checked.

The CVO also manages the upgrade process. CVO orchestrates desired state changes to each of the Operators in a specific order, with constant health checking along the way. During the upgrade process, the Machine Config Operator (MCO) applies the new configuration to your cluster nodes. The MCO cordons the number of nodes as specified in the machine configuration pool and marks them as unavailable. The MCO then applies the new configuration and reboots the machine. The control plane and etcd are upgraded first, then the OS and config of the nodes that run the control plane, and finally, the rest of the cluster Operators. Upgrading a three nodes Control Plane usually takes about an hour.

Worker nodes are upgraded after the Control Plane has finished upgrading. These do not block the cluster’s upgrade process, because nodes may come and go as autoscaling takes place.

Terminology

Channels represented the appropriate release versions for cluster upgrades. Upgrade channels allow you to choose an upgrade strategy. Upgrade channels are tied to a minor version of OpenShift Container Platform. OpenShift 4.6 offers the following upgrade channels:

Stable: This channel contains releases as soon as their errata are published, releases are added to the stable channel after a delay of several hours to a day.
Fast: This channel is updated with new versions as soon as Red Hat declares the given version as a general availability release.
Candidate: Release candidates contain all the features of the product but are not supported. Use release candidate versions to test feature acceptance and assist in qualifying the next version of OpenShift Container Platform.
EUS: This channel certain minor versions of OpenShift Container Platform offer an Extended Update Support (EUS). These EUS versions extend the maintenance phase for customers with Premium Subscriptions to 14 months.

Upgrade path provided a channel graph to plan an upgrade from one version to another. There are two ways you can determine the upgrade path.

OpenShift Administrator Console: You need admin privileges to access to the web console. From the web console, click Administration → Cluster Settings and review the contents of the Details tab. In this example, current version is 4.6.38, stable channel 4.6 and some upgrades available. The minor target version is 4.6.42. With changing the channel to stable-4.7, major target version is the latest stable 4.7 version.
RedHat Cloud Web Interface can be access from https://access.redhat.com/labs/ocpupgradegraph/update_path. Depends on the provided current, target channel and OpenShift version on the RedHat Cloud Web interface, it will build upgrade path. It generates the update steps need from OpenShift CLI or Web console.

OpenShift Administrator Console or RedHat Cloud Web Interface displays all valid update paths. Do not force an update to a version that upgrade path does not suggest.

Only upgrading to a newer version is supported. Reverting or rolling back your cluster to a previous version is not supported. If your upgrade fails, contact Red Hat support.

Prerequisites

You have access to the cluster as a user with “admin” privileges to perform the upgrade.
Take a recent etcd backup in case your upgrade fails and need to restore your cluster to a previous state. Only save a backup from a single master node.

### Start a debug session for a control plane node:      
oc debug node/<node_name>### Change your root directory to the host:   
chroot /host### Run the backup script: 
/usr/local/bin/cluster-backup.sh /home/core/assets/backup

Your cluster must be healthy to perform an upgrade.

Check that all nodes are in a Ready status and using same version (oc get nodes).
No node in degraded state (oc get mcp).
All pods are up and running (oc get pods -A | egrep -v ‘Running|Completed|Terminated|Succeeded’).
Verify current status version is available and there is no upgrade in progress (oc get clusterversion).
Ensure all operators are available, none of them should be degraded (oc get co).
Ensure that all machine config pools (MCPs) are running and not paused. Nodes associated with a paused MCP are skipped during the update process (oc get mcp).
Confirm latest updates available for the cluster (oc adm upgrade).

The Upgrade

Upgrade can be performed from OpenShift Administrator Console or using OpenShift CLI.

OpenShift Administrator Console — The over-the-air OpenShift 4,x upgrade is as simple as one-click operation. From the web console, click Administration → Cluster Settings and choose the Details tab.

Make sure channel is set to the correct channel for the version that you want to update to. Then click on the Upgrade button.

OpenShift CLI — Follow the step if you are preferring to perform upgrade from OpenShift CLI.

1. Log in to OpenShift cluster from bastion node:

$ oc login OpenShift_URL:port

2. If you are upgrading to a next channel, set the correct channel for the version that you want to update to. In this example, updating next channel to 4.7, as current channel is 4.6:

$ oc patch clusterversion version --type="merge" -p '{"spec":{"channel":"stable-4.7"}}'

3. Upgrade to OpenShift 4.7.23 or the latest stable version on 4.7:

$ oc adm upgrade --to="4.7.23"

$ oc adm upgrade --to-latest

4. Monitor the upgrade:

$ watch -n 10 "oc adm upgrade && oc get co && oc get nodes -o wide"

Troubleshooting

Check the current status, version and if upgrade is in progress.

$ oc get clusterversion

The yaml output will show status messages about your upgrade.

$ oc get clusterversion -o yaml

Review the logs from the Cluster Version Operator (or CVO), as it should tell you how it’s progressing; in more detail than the events and or status updates seen by the clusterversion object.

$ oc logs $(oc get pod -n openshift-cluster-version -l k8s-app=cluster-version-operator -oname) -n openshift-cluster-version

Conclusion

Any kind of system or application upgrade is a challenging operation. But with careful and strategic plan, upgrade can be complete smoothly. OpenShift 4 brings supreme automation to cluster scaling, maintenance, and upgrade. You just need few clicks and cluster does the rest. OpenShift upgrade is truly a one-click reality.