Upgrade to v4.10
This guide lists breaking changes for v4.10 and also shows you how to move to the new, more efficient Intake Head architecture.
Breaking Changes and Compatibility Notices
Unified Authentication is Default
If you're upgrading from version 4.8 or earlier, note that this version of Hydrolix uses Unified Authentication by default, which is a breaking change for existing customers using Basic Authentication. To maintain the current behavior when upgrading a cluster from 4.8 or earlier, ensure that unified_auth: false
is set in your cluster's configuration.
Invitation URL API Change
The /config/v1/inviteurl
API endpoint has been changed to /config/v1/invites/invite_url
. Refer to the API documentation to check this and other invitation system API changes.
Continuous Control Loop Operator
Rather than running just once per deployment, the operator now runs continuously. Temporary manual changes to deployments will now be overwritten by this control loop. When making manual changes, scale the operator to 0 with kubectl scale --replicas 0 deployment/operator
.
Rolling Back to 4.8 with Batch Ingest
If you are using batch ingest, and need to roll back to 4.8.x after upgrading to this version, contact Hydrolix Customer Success before rolling back.
Recent Versions of PostgreSQL
This version of Hydrolix requires PostgreSQL version 11 or higher. We strongly recommend you use PostgreSQL 13 or higher for easy upgrading. If you are using PostgreSQL version 11 or 12, enable the ltree
extension as superuser, and consider upgrading to an up-to-date version of PostgreSQL.
More Restrictive RBAC
RBAC Version 2, introduced in Hydrolix version 4.6.5, has tighter security than RBAC Version 1. In particular, read-only roles have more limited visibility of the system.
Upgrade to the Ingest Head Architecture
The new 4.10 release contains a more efficient and stable architecture for HTTP streaming ingest. This new architecture is not the default, so follow this guide to put it in place. Once done, you will have replaced your stream-head
and stream-peer
pods with the new intake-head
pod and removed redpanda
from your intake pipeline.
You can find a comparison of the two architectures on the Ingest page.
This doesn't affect Kafka ingest
Even though this architecture reduces the Hydrolix cluster's internal dependence on Redpanda's Kafka queues, it doesn't affect how
kafka-peer
s work. Any Kafka-related ingest you have configured will be unaffected by this change.
This guide assumes you have kubectl
set up properly and pointed at your Hydrolix cluster.
1. Upgrade your cluster
Upgrade to the latest version of Hydrolix as outlined in the Release Notes and in Upgrade Hydrolix For example, on GKE:
kubectl apply -f "https://www.hydrolix.io/operator/v4.10.7/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"
...and on EKS:
kubectl apply -f "https://www.hydrolix.io/operator/v4.10.7/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"
Replace the version number with what you find in the Release Notes. This won't immediately change your intake pipeline, but it will put the new intake-head
service in place for you to activate.
2. Scale up the new intake-head
intake-head
Create intake-head
pods by modifying your scale configuration (without HPA):
yaml
scale:
intake-head:
replicas: 2
Wait for the intake-head
s to be in the running state. Check their logs to verify with k9s
or with a kubectl
command:
kubectl logs <pod name>
3. Shift all traffic to the intake-head
intake-head
Change the scale for stream-head
to 0 replicas, but keep the stream-peer
running as before to finish reading data from the Redpanda queue. Also, update the intake-head
config to use HPA:
scale:
stream-head:
replicas: 0
stream-peer:
replicas: 1-100
intake-head:
replicas: 2-100
Note that your values will probably differ from those above.
This will scale up the intake-head
and keep the stream-peer
running.
Wait for the stream-peer
to finish reading all the data from Redpanda. To make sure it's finished, check its logs to make sure your Redpanda instances have stopped pulling ingest data. Use k9s
or kubectl
:
kubectl logs redpanda-0
4. Scale down stream-peers
stream-peers
Once stream-peer
has finished reading the Redpanda queue, scale down the stream-peer
and reduce your redpanda
deployment. For example:
scale:
stream-head:
replicas: 0
stream-peer:
replicas: 0
intake-head:
replicas: 2-100
redpanda:
cpu: 2
data_storage: 100Gi
memory: 4Gi
replicas: 1
Since you're changing the statefulset and PVC of the Redpanda deployment, remove the current statefulset PVC to let the operator re-create our new smaller one:
kubectl delete pvc data-redpanda-0
kubectl delete pvc data-redpanda-1
kubectl delete pvc data-redpanda-2
kubectl delete pvc data-redpanda-3
kubectl delete pvc data-redpanda-4
kubectl delete pvc data-redpanda-5
kubectl delete statefulset redpanda
There will be more or fewer commands, depending on how many redpanda
instances you had configured.
The operator will re-create a new, smaller "redpanda-0" instance which will only be used for the Hydrolix logging.
You now have a simplified, more efficient cluster!
Notes
- If you should decide to roll back to v4.8 and you're using batch ingest, extra care is needed. Contact Hydrolix Customer Success to make sure it happens without incident.
- Even an unused Redpanda instance will allocate all the memory it can for performance reasons. Make sure you scale Redpanda down to save cloud costs.
Updated about 2 months ago