Hydrolix ACME Client

Overview

Hydrolix includes software for automatic certificate acquisition and renewal.

This is the default and recommended option for managing TLS in a Hydrolix cluster. See Enable TLS for more TLS options.

The TLS industry has adopted a standard called Automatic Certificate Management Environment (ACME) which can be supported by a Certification Authority (CA).

This page describes how to use the ACME client in a Hydrolix cluster for automated provisioning of TLS certificates.

The Hydrolix implementation automates sourcing certificates from the Let's Encrypt CA using the Lego project.

Prerequisites

Before you can start this guide, you'll need the following:

• A deployed Hydrolix cluster
• A publicly reachable IP address on the cluster
• An active DNS record for the cluster's name

Generate a certificate in Hydrolix

1. Remove the traefik-tls secret before proceeding. To prevent loss of an existing certificate, the Hydrolix ACME client won't overwrite an existing traefik-tls secret.
2. Double-check your hydrolix_url for spelling errors, ensuring that you can look up the name in the DNS.
3. Set the configuration option acme_enabled to true in hydrolixcluster.yaml.
4. Load the configuration changes to your Hydrolix cluster.
5. Hydrolix automatically generates a certificate for your cluster and stores it in a Kubernetes secret named traefik-tls.

Let's Encrypt with an open allowlist

Hydrolix selects the Let's Encrypt CA if your cluster allows inbound access to 0.0.0.0/0 in the ip_allowlist.

Certificates are valid for 90 days. Hydrolix checks weekly for expiration and renews early.

Automatic renewal

Hydrolix runs an acme-renewal job which attempts to renew the certificate if the remaining time before expiration is less than 37 days.

Reminder notifications

Hydrolix doesn't provide any email notifications of certificate expiration. CAs usually send courtesy mesasges to the admin_email used for acquiring the certificate. This email address should receive notifications of upcoming expiration directly from the CA.

Confirm certificate generation

To confirm certificate deployment, run the following command to view details of the certificate.

kubectl -n <namespace> get secret traefik-tls -o yaml

For example:

$ kubectl -n hydrolix get secret traefik-tls -o yaml
apiVersion: v1
data:
  tls.crt: someCertificateData
  tls.key: someCertificateData
kind: Secret
metadata:
  creationTimestamp: "2023-06-27T15:27:12Z"
  name: traefik-tls
  namespace: hydrolix
  resourceVersion: "32029846"
  uid: 33c06aa2-b81f-44a9-9123-c24e1af94bb5
type: kubernetes.io/tls

See also Enable TLS.

Troubleshoot

There are some specific troubleshooting steps when using the Hydrolix acme-client and also common Certificate Troubleshooting tips.

1. Check the hostname for spelling errors and correct any mistakes.
2. Verify a successful DNS lookup to one or more DNS resolvers for the hostname.
3. Confirm from your workstation with the host $HDX_HOSTNAME command; if it fails, confirm that the authoritative DNS entry is present.
4. Confirm using a public resolver with the command host $HDX_HOSTNAME 8.8.8.8. If it fails, diagnose and resolve the authoritative DNS issue.
5. Confirm the init-acme job has run; see Verify the cluster job runs.
6. Confirm the acme-renewal job is scheduled; see Verify Kubernetes cron job runs.

7. Confirm that there is a reachable HTTP service inside the Hydrolix cluster:

   curl --silent -v -o /dev/null --fail "http://$HDX_HOSTNAME"
   

   If this fails, see Verify startup.

The certificate init-acme and acme-renewal jobs both retry upon failure up to six (6) times before stopping. This isn't configurable.

Verify the cluster job runs

When the Hydrolix ACME client is enabled, the job init-acme is started. To find the init job, run:

$ kubectl -n <namespace> get job --field-selector status.successful=1 | grep acme
init-acme-509c50f0              1/1           21s        24h

This job may disappear upon success or if the service has been running for more than a day. Hydrolix clears successful job notifications automatically.

Verify Kubernetes cron job runs

The acme-renewal cron job, created at startup, routinely runs, checks if renewal is required and performs the renewal operations with the ACME provider if necessary.

To verify, find the acme-renewal job:

$ kubectl -n <namespace> get cronjob
NAME                SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
acme-renewal        0 2 * * SUN   False     0        <none>          24h

Verify startup

List the pods to locate the identifier of the init-acme job.

kubectl -n <namespace> get pods

With the pod identifier, examine the logs:

kubectl -n <namespace> logs init-acme-509c50f0-2zghd

An example of successful initialization:

Attempt 1/300: Routing not ready, retrying in 1s...
External routing confirmed working! Ready to proceed with ACME challenge.

Requesting a new certificate
2025/06/03 17:28:47 No key found for account username@example.com. Generating a P256 key.
  [ ... ]

Consider IP allowlist

There are several reasons for unsuccessful initialization, but one failure scenario involves a conflict between the use of an IP allowlist and the ACME provider's verification steps. The symptom appears in the logs of the init-acme job as a Routing not ready error.

Testing external routing via ${myhost}.hydrolix.live...
Attempt 1/300: Routing not ready, retrying in 1s...
 [ ... ]
Attempt 300/300: Routing not ready, retrying in 1s...
External routing test failed after 300 attempts.

The above failure indicates that the test client can't reach the configured hostname in the Hydrolix spec hydrolix_url.

Traffic from the test client leaves the cluster completely, returning as ingress traffic, in order to simulate an incoming ACME challenge request.

After confirming again that the cluster hostname in the hydrolix_url is a publicly visible and valid DNS name, consider temporarily adding 0.0.0.0/0 to the ip_allowlist. See Configure IP Access.

Remove certificate and jobs

To wipe the slate clean and start over.

1. Remove the acme-account and traefik-tls secrets.

   kubectl -n <namespace> delete secret traefik-tls
   kubectl -n <namespace> delete secret acme-account
   

2. Remove the init-acme job.

   kubectl -n <namespace> delete job init-acme-XYZ  # -- find exact name running in cluster
   

3. Remove the acme-renewal cronjob.

   kubectl -n <namespace> delete cronjob acme-renewal
   

Hydrolix-generated certificates update automatically

If you're having issues with renewal, please contact Hydrolix Support.