Downgrade instructions from v5.7.4

Overview

The v5.7.x release included an upgrade to Keycloak v26.3.0.

Use the special downgrade procedure only for deployments where

• Keycloak runs on a single node
• The Hydrolix operator is prohibited from running multiple Keycloak pods on that node

For all other clusters, use the standard downgrade procedure, v5.7.4 to v5.6.2.

Standard downgrade procedure, v5.7.4 to v5.6.2

Outline

1. Perform the schema migration rollback
2. Perform the downgrade to v5.6.2

Perform the schema migration rollback

The following steps restore the schema to the schema expected by v5.6.2.

1. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.

2. Select the turbine-api pod and turbine-api container.

3. Invoke a shell by entering s.

4. Execute each of the following commands, which will produce STDERR to terminal. Each must exit cleanly.

   Shell

   ./manage.py release_5_6
   

5. Exit the k9s utility by pressing Ctrl-C.

Perform the standard downgrade to v5.6.2

The below instructions are the same as the upgrade instructions from the v5.6.2 release notes.

If you have a self-managed installation, apply the new operator directly with the kubectl command examples below. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

GKE

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

EKS

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

LKE and AKS

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}"

Monitor the standard downgrade process to v5.6.2

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

Special downgrade procedure v5.7.4 to v5.6.2

Use these instructions only for single-node clusters where operator is configured to forbid multiple Keycloak pods. This deployment configuration is not unusual for development clusters.

During this downgrade process, the cluster will not be fully usable.

Outline of downgrade steps

1. Execute the schema migration rollback
2. Scale down keycloak
3. Upgrade: Apply the new Hydrolix operator
4. Scale up keycloak
5. Monitor the downgrade process

Execute the schema migration rollback

The following steps restore the schema to the schema expected by v5.6.2.

1. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.

2. Select the turbine-api pod and turbine-api container.

3. Invoke a shell by entering s.

4. Execute each of the following commands, which will produce STDERR to terminal. Each must exit cleanly.

   Shell

   ./manage.py release_5_6
   

5. Exit the k9s utility by pressing Ctrl-C.

Scale down Keycloak (downgrade)

Stop the Keycloak pods.

kubectl scale deployment -n "${HDX_KUBERNETES_NAMESPACE}" keycloak --replicas=0

Use your preferred Kubernetes monitoring tool to confirm the keycloak pod terminates and does not restart.

Perform the downgrade to v5.6.2

The below instructions are the same as the upgrade instructions from the v5.6.2 release notes.

If you have a self-managed installation, apply the new operator directly with the kubectl command examples below. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

GKE

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

EKS

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

LKE and AKS

kubectl apply -f "https://www.hydrolix.io/operator/v5.6.2/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}"

Scale up Keycloak (downgrade)

Scale Keycloak back up. Replace the replica count with your desired number of pods. If you are using these instructions, it was probably 1.

kubectl scale deployment -n "${HDX_KUBERNETES_NAMESPACE}" keycloak --replicas=1

Use your preferred Kubernetes monitoring tool to observe that the keycloak pod restarts.

Monitor the downgrade process

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

If you need help, contact Hydrolix Support.