Skip to content

Upgrade to v4.10

Do not skip versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each version.

Breaking Changes and Compatibility Notices⚓︎

Unified Authentication is Default⚓︎

If you're upgrading from version 4.8 or earlier, note that this version of Hydrolix uses Unified Authentication by default, which is a breaking change for existing customers using Basic Authentication. To maintain the current behavior when upgrading a cluster from 4.8 or earlier, ensure that unified_auth: false is set in your cluster's configuration.

Invitation URL API Change⚓︎

The /config/v1/inviteurl API endpoint has been changed to /config/v1/invites/invite_url. Refer to the API documentation to check this and other invitation system API changes.

Continuous Control Loop Operator⚓︎

Rather than running just once per deployment, the operator now runs continuously. Temporary manual changes to deployments will now be overwritten by this control loop. When making manual changes, scale the operator to 0 with kubectl scale --replicas 0 deployment/operator.

Rolling Back to 4.8 with Batch Ingest⚓︎

If you are using batch ingest, and need to roll back to 4.8.x after upgrading to this version, contact Hydrolix Customer Success before rolling back.

Recent Versions of PostgreSQL⚓︎

This version of Hydrolix requires PostgreSQL version 11 or higher. We strongly recommend you use PostgreSQL 13 or higher for easy upgrading. If you are using PostgreSQL version 11 or 12, enable the ltree extension as superuser, and consider upgrading to an up-to-date version of PostgreSQL.

More Restrictive RBAC⚓︎

RBAC Version 2, introduced in Hydrolix version 4.6.5, has tighter security than RBAC Version 1. In particular, read-only roles have more limited visibility of the system.

Upgrade to the Ingest Head Architecture⚓︎

The new 4.10 release contains a more efficient and stable architecture for HTTP streaming ingest. This new architecture is not the default, so follow this guide to put it in place. Once done, you will have replaced your stream-head and stream-peer pods with the new intake-head pod and removed redpanda from your intake pipeline.

You can find a comparison of the two architectures on the Ingest page.

This doesn't affect Kafka ingest

Even though this architecture reduces the Hydrolix cluster's internal dependence on Redpanda's Kafka queues, it doesn't affect how kafka-peers work. Any Kafka-related ingest you have configured will be unaffected by this change.

This guide assumes you have kubectl set up properly and pointed at your Hydrolix cluster.

1. Upgrade your cluster⚓︎

Upgrade to the latest version of Hydrolix as outlined in the Release Notes and in Upgrade Hydrolix For example, on GKE:

1
kubectl apply -f "https://www.hydrolix.io/operator/v4.10.7/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

...and on EKS:

1
kubectl apply -f "https://www.hydrolix.io/operator/v4.10.7/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

Replace the version number with what you find in the Release Notes. This won't immediately change your intake pipeline, but it will put the new intake-head service in place for you to activate.

2. Scale up the new intake-head⚓︎

Create intake-head pods by modifying your scale configuration (without HPA):

1
2
3
4
yaml
    scale:
      intake-head:
        replicas: 2

Wait for the intake-heads to be in the running state. Check their logs to verify with k9s or with a kubectl command:

1
kubectl logs <pod name>

3. Shift all traffic to the intake-head⚓︎

Change the scale for stream-head to 0 replicas, but keep the stream-peer running as before to finish reading data from the Redpanda queue. Also, update the intake-head config to use HPA:

1
2
3
4
5
6
7
    scale:
      stream-head:
        replicas: 0
      stream-peer:
        replicas: 1-100
      intake-head:
        replicas: 2-100

Note that your values will probably differ from those above.

This will scale up the intake-head and keep the stream-peer running.

Wait for the stream-peer to finish reading all the data from Redpanda. To make sure it's finished, check its logs to make sure your Redpanda instances have stopped pulling ingest data. Use k9s or kubectl:

1
kubectl logs redpanda-0

4. Scale down stream-peers⚓︎

Once stream-peer has finished reading the Redpanda queue, scale down the stream-peer and reduce your redpanda deployment. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    scale:
      stream-head:
        replicas: 0
      stream-peer:
        replicas: 0
      intake-head:
        replicas: 2-100
      redpanda:
        cpu: 2
        data_storage: 100Gi
        memory: 4Gi
        replicas: 1

Since you're changing the statefulset and PVC of the Redpanda deployment, remove the current statefulset PVC to let the operator re-create our new smaller one:

1
2
3
4
5
6
7
kubectl delete pvc data-redpanda-0
kubectl delete pvc data-redpanda-1
kubectl delete pvc data-redpanda-2
kubectl delete pvc data-redpanda-3
kubectl delete pvc data-redpanda-4
kubectl delete pvc data-redpanda-5
kubectl delete statefulset redpanda

There will be more or fewer commands, depending on how many redpanda instances you had configured.

The operator will re-create a new, smaller "redpanda-0" instance which will only be used for the Hydrolix logging.

You now have a simplified, more efficient cluster!

Notes⚓︎

  • If you should decide to roll back to v4.8 and you're using batch ingest, extra care is needed. Contact Hydrolix Customer Success to make sure it happens without incident.
  • Even an unused Redpanda instance will allocate all the memory it can for performance reasons. Make sure you scale Redpanda down to save cloud costs.