Skip to content

Upgrade to v5.2

Do not skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version.

Example:
Upgrade from 5.5.0 → 5.6.x → 5.7.4, not 5.5.0 → 5.7.4.

Overview⚓︎

The v5.2.x release introduces significant infrastructure changes, notably the removal of Redpanda dependency in favor of Vector for log ingestion.

Key changes in v5.2.x⚓︎

  • Redpanda removal: Vector now handles log ingestion through a new hydrolix-intake-head pool instead of Redpanda
  • New hydrolix pool: A static pool named hydrolix-intake-head is created for Vector log ingestion
  • Kafka source deletion: The old Kafka-based hydro.logs source is removed during upgrade
  • Pulumi stack sync: This release may require syncing of Pulumi stacks due to the new hydrologs pool

Redpanda cleanup

The upgrade process does not automatically scale down or delete the Redpanda StatefulSet. After verifying the upgrade is successful, manually scale down Redpanda to confirm no dependencies remain: kubectl scale statefulset redpanda -n ${HDX_KUBERNETES_NAMESPACE} --replicas=0

Upgrade instructions⚓︎

Prerequisites for EKS deployments⚓︎

For EKS deployments, ensure the following tunable is set in your Hydrolix spec before upgrading:

1
db_bucket_credentials_method: static

Standard upgrade procedure⚓︎

Select the appropriate command based on your cloud provider. Replace the "x" in "v5.2.x" with the latest patch release of v5.2 available:

GKE upgrade⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.2.x/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

EKS upgrade⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.2.x/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

LKE and AKS upgrade⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.2.x/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}"

Post-upgrade verification⚓︎

After the upgrade is complete:

  1. Verify the new hydrolix-intake-head pool is created and running:
1
kubectl get pools -n ${HDX_KUBERNETES_NAMESPACE}
  1. Confirm Vector is ingesting logs successfully:
1
kubectl logs -n ${HDX_KUBERNETES_NAMESPACE} -l app=vector

  1. Check that the old Kafka source has been removed from your configuration.

  2. Optionally, scale down Redpanda to verify no dependencies remain.

v5.2.x to v5.1.x rollback procedure⚓︎

Rollback overview⚓︎

Rolling back from v5.2.x to v5.1.x requires regenerating the Kafka-peer-based hydro.logs pool using Redpanda. Before you perform the downgrade, delete stale Redpanda data.

Critical: Delete Redpanda PVC before rollback

The rollback regenerates the hydro.logs pool using Kafka and Redpanda. You must delete the Redpanda PersistentVolumeClaim (PVC) before rolling back to avoid retaining stale topic data that could cause issues.

Phase 1: Delete Redpanda PersistentVolumeClaim⚓︎

Remove the Redpanda PVC before rollback to ensure a clean state:

  1. List Redpanda PVCs:
1
kubectl get pvc -n ${HDX_KUBERNETES_NAMESPACE} -l app=redpanda
  1. Delete each Redpanda PVC:
1
kubectl delete pvc -n ${HDX_KUBERNETES_NAMESPACE} -l app=redpanda
  1. Verify PVCs are deleted:
1
kubectl get pvc -n ${HDX_KUBERNETES_NAMESPACE} -l app=redpanda

Phase 2: Perform the rollback⚓︎

After deleting the Redpanda PVCs, apply the v5.1.x operator. Select the appropriate command based on your cloud provider:

GKE rollback⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.1.3/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

EKS rollback⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.1.3/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

LKE and AKS rollback⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v5.1.3/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}"

Phase 3: Verify rollback completion⚓︎

After the rollback is complete:

  1. Verify Redpanda pods are running:
1
kubectl get pods -n ${HDX_KUBERNETES_NAMESPACE} -l app=redpanda
  1. Confirm the hydro.logs pool has been regenerated with Kafka/Redpanda:
1
kubectl get pools -n ${HDX_KUBERNETES_NAMESPACE}
  1. Check Redpanda logs for successful startup:
1
kubectl logs -n ${HDX_KUBERNETES_NAMESPACE} -l app=redpanda
  1. Verify the Hydrolix cluster is functioning through the UI and API.

Troubleshooting⚓︎

If you encounter issues during the rollback:

  • Redpanda pods fail to start: Check that PVC deletion completed successfully and verify storage class availability.
  • hydro.logs pool not recreated: Examine init-cluster and init-turbine-api job logs:
1
2
kubectl logs -l app=init-cluster -n ${HDX_KUBERNETES_NAMESPACE}
kubectl logs -l app=init-turbine-api -n ${HDX_KUBERNETES_NAMESPACE}
  • Vector still running: The Vector deployment should automatically be removed during rollback, but verify manually if needed:
1
kubectl get deployments -n ${HDX_KUBERNETES_NAMESPACE} -l app=vector

Contact Hydrolix Support at mailto:support@hydrolix.io for additional assistance.