Skip to content

Independent Container Versioning

Overview⚓︎

As of v5.7.0, Hydrolix clusters support independent deployment versions for individual components.

This feature allows you to override container images and tags per component, enabling granular version control and easier testing, rollback, or phased upgrades without affecting the rest of the cluster.

Use independent component versioning to:

  • Safely test updates to specific services without redeploying the entire cluster.
  • Pin or roll back a component to a stable version while other components continue to evolve.

When upgrading a Hydrolix cluster, pods will not be re-deployed if all of their containers have their image and version overridden and their values haven't changed.

Configuration⚓︎

Enable this feature in the hdx configuration spec using the containers key. When a container image override is set, all matching containers, regardless of workload type (Deployment, StatefulSet, Job, or DaemonSet), use the specified image and tag.

```yaml Example image override config spec: containers: merge-controller: image: example.com/merge-controller tag: "v5.6.0" turbine-api: tag: v5.7.0 elasticsearch: image: example.com/elasticsearch 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
In the example above:

- The `merge-controller` image is from a custom registry (`example.com`) and sets a custom version tag.
- The `turbine-api` container uses the default registry (`us-docker.pkg.dev/hdx-art/t`) but overrides the version tag.
- The `elasticsearch` container is from a custom registry (`example.com`). Because a version isn't explicitly set, it deploys with the version from the operator manifest. See [Version resolution](#version-resolution) for more information.
- All workloads referencing these container names will be automatically updated with the new image and tag.

### Image resolution rules

If the `image` key contains a slash (`/`), Hydrolix interprets the path as:

- **Registry path**: everything before the last `/`
- **Image name**: the final segment after the last `/`

If the `image` key doesn't contain a slash, the registry path defaults to:
us-docker.pkg.dev/hdx-art/t 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
### Version resolution

When using the default registry, use `version` tags which correspond to a published [Hydrolix cluster release version](../../release-notes/v5.7.4.md).

If an image is set without a version tag, the component version will default to component image in the operator manifest. For example, a Hydrolix cluster deployed with a `v5.7.0` operator will deploy components by default that are also `v5.7.0`. You can verify the image version in the operator manifest directly by visiting `https://hdx-infrastructure.s3.us-east-2.amazonaws.com/v/manifest-$VERSION.json`.

For example, a Hydrolix cluster running version `v5.7.0` downloads and defaults to component images specified in manifest `https://hdx-infrastructure.s3.us-east-2.amazonaws.com/v/manifest-v5.7.0.json`. Versions indicated in the manifest (for example: `"MERGE_CONTROLLER_VERSION": "db7063e7"`) are identified by `Tags` in the [default image repository](https://us-docker.pkg.dev/hdx-art/t) (Filter: `Tags: db7063e7`)

## Monitor overrides with Prometheus

Prometheus metrics for image overrides require deploying a [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) container with the Hydrolix cluster. This container cannot be deployed through the Hydrolix operator and must instead be manually deployed and managed.

To track image overrides using Prometheus, update the deployment configuration for the `kube-state-metrics` container.

### Add annotation flags to generate image override metric

Include the following argument in the `kube-state-metrics` container spec:

```yaml Config to enable image override metric
spec:
  containers:
  - args:
    - --metric-annotations-allowlist=pods=[hydrolix.io/image-override]
    - --metric-allowlist=kube_pod_annotations,kube_pod_container_status_restarts_total,kube_pod_container_status_last_terminated_reason,kube_namespace_created,kube_pod_container_resource_limits

metric-annotations-allowlist: This arg enables the image override reporting.
metric-allowlist: This arg ensures Hydrolix generates the annotation_hydrolix_io_image_override metric which is queried in Prometheus.

This ensures only the relevant annotation is captured, avoiding excessive metric cardinality and annotation size issues.

Query overrides using PromQL⚓︎

You can identify all pods with image overrides in Prometheus using:

```text Promql example query kube_pod_annotations{annotation_hydrolix_io_image_override != ""} 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
```text Promql example output
kube_pod_annotations{
annotation_hydrolix_io_image_override="turbine", 
annotation_hydrolix_io_service="merge-peer", 
app="kube-state-metrics", 
deployment_id="example-cluster", 
exported_namespace="example-namespace", 
instance="10.2.3.164:8080", 
job="presence", 
namespace="example-namespace", 
pod="merge-peer-iii-5cbb8c794b-6b5qf", 
port_name="kube-state-metrics", 
service="kube-state-metrics", 
uid="17ee4f19-b25c-43e3-ab66-67f972a4f6b8"}