Independent Container Versioning

📘

Feature introduced in v5.7.4

Overview

As of v5.7.4, Hydrolix clusters support independent deployment versions for individual components.

This feature allows you to override container images and tags per component, enabling granular version control and easier testing, rollback, or phased upgrades without affecting the rest of the cluster.

Use independent component versioning to:

• Safely test updates to specific services without redeploying the entire cluster.
• Pin or roll back a component to a stable version while other components continue to evolve.

When upgrading a Hydrolix cluster, pods will not be re-deployed if all of their containers have their image and version overridden and their values haven't changed.

Configuration

Enable this feature in the hdx configuration spec using the containers key. When a container image override is set, all matching containers, regardless of workload type (Deployment, StatefulSet, Job, or DaemonSet), use the specified image and tag.

spec:
  containers:
    merge-controller: 
      image: example.com/merge-controller
      tag: "v5.6.0"
    turbine-api:
      tag: v5.7.4
    elasticsearch: 
      image: example.com/elasticsearch

In the example above:

• The merge-controller image is from a custom registry (example.com) and sets a custom version tag.
• The turbine-api container uses the default registry (us-docker.pkg.dev/hdx-art/t) but overrides the version tag.
• The elasticsearch container is from a custom registry (example.com). Because a version isn't explicitly set, it deploys with the version from the operator manifest. See Version resolution for more information.
• All workloads referencing these container names will be automatically updated with the new image and tag.

Image resolution rules

If the image key contains a slash (/), Hydrolix interprets the path as:

• Registry path: everything before the last /
• Image name: the final segment after the last /

If the image key doesn't contain a slash, the registry path defaults to:

us-docker.pkg.dev/hdx-art/t

Version resolution

When using the default registry, use version tags which correspond to a published Hydrolix cluster release version.

If an image is set without a version tag, the component version will default to component image in the operator manifest. For example, a Hydrolix cluster deployed with a v5.7.4 operator will deploy components by default that are also v5.7.4. You can verify the image version in the operator manifest directly by visiting https://hdx-infrastructure.s3.us-east-2.amazonaws.com/v/manifest-$VERSION.json.

For example, a Hydrolix cluster running version v5.7.4 downloads and defaults to component images specified in manifest https://hdx-infrastructure.s3.us-east-2.amazonaws.com/v/manifest-v5.7.4.json. Versions indicated in the manifest (for example: "MERGE_CONTROLLER_VERSION": "db7063e7") are identified by Tags in the default image repository (Filter: Tags: db7063e7)

Monitoring overrides with Prometheus

Prometheus metrics for image overrides require deploying a kube-state-metrics container with the Hydrolix cluster. This container cannot be deployed via the Hydrolix operator and must instead be manually deployed and managed.

To track image overrides using Prometheus, update the deployment configuration for the kube-state-metrics container.

Add annotation flags to generate image override metric

Include the following argument in the kube-state-metrics container spec:

spec:
  containers:
  - args:
    - --metric-annotations-allowlist=pods=[hydrolix.io/image-override]
    - --metric-allowlist=kube_pod_annotations,kube_pod_container_status_restarts_total,kube_pod_container_status_last_terminated_reason,kube_namespace_created,kube_pod_container_resource_limits

metric-annotations-allowlist: This arg enables the image override reporting.
metric-allowlist: This arg ensures Hydrolix generates the annotation_hydrolix_io_image_override metric which is queried in Prometheus.

This ensures only the relevant annotation is captured, avoiding excessive metric cardinality and annotation size issues.

Query overrides using PromQL

You can identify all pods with image overrides in Prometheus using:

kube_pod_annotations{annotation_hydrolix_io_image_override != ""}

kube_pod_annotations{
annotation_hydrolix_io_image_override="turbine", 
annotation_hydrolix_io_service="merge-peer", 
app="kube-state-metrics", 
deployment_id="example-cluster", 
exported_namespace="example-namespace", 
instance="10.2.3.164:8080", 
job="presence", 
namespace="example-namespace", 
pod="merge-peer-iii-5cbb8c794b-6b5qf", 
port_name="kube-state-metrics", 
service="kube-state-metrics", 
uid="17ee4f19-b25c-43e3-ab66-67f972a4f6b8"}