Hydrolix ACME Client

Overview

Hydrolix includes software for automatic certificate acquisition and renewal.

This is the default and recommended option for managing TLS in a Hydrolix cluster. See Enable TLS for more TLS options.

The TLS industry has adopted a standard called Automatic Certificate Management Environment (ACME) which can be supported by a Certification Authority (CA).

This page describes how to use the ACME client in a Hydrolix cluster for automated provisioning of TLS certificates.

The Hydrolix implementation automates sourcing certificates from either the Let's Encrypt or Buypass CAs using the Lego project.

Prerequisites

Before you can start this guide, you'll need the following:

• A deployed Hydrolix cluster
• A publicly reachable IP address on the cluster
• An active DNS record for the cluster's name

Generate a certificate in Hydrolix

1. Remove the traefik-tls secret before proceeding. To prevent loss of an existing certificate, the Hydrolix ACME client won't overwrite an existing traefik-tls secret.
2. Double-check your hydrolix_url for spelling errors, ensuring that you can look up the name in the DNS.
3. Set the configuration option acme_enabled to true in hydrolixcluster.yaml.
4. Load the configuration changes to your Hydrolix cluster.
5. Hydrolix automatically generates a certificate for your cluster and stores it in a Kubernetes secret named traefik-tls.

Let's Encrypt with an open allowlist

Hydrolix selects the Let's Encrypt CA if your cluster allows inbound access to 0.0.0.0/0 in the ip_allowlist.

Certificates are valid for 90 days. Hydrolix checks weekly for expiration and renews early.

Buypass with a tighter allowlist

Hydrolix selects the Buypass CA and appends its IP addresses to the ip_allowlist if your cluster has anything other than 0.0.0.0/0 in its ip_allowlist.

Hydrolix automatically inserts the addresses into the cluster configuration. View the Hydrolix configuration with kubectl describe hdx to verify that the correct IP addresses are present.

Automatic renewal

Hydrolix runs an acme-renewal job which attempts to renew the certificate if the remaining time before expiration is less than 37 days.

Reminder notifications

Hydrolix doesn't provide any email notifications of certificate expiration. CAs usually send courtesy mesasges to the admin_email used for acquiring the certificate. This email address should receive notifications of upcoming expiration directly from the CA.

Confirm certificate generation

To confirm certificate deployment, run the following command to view details of the certificate.

kubectl -n <namespace> get secret traefik-tls -o yaml

For example:

$ kubectl -n hydrolix get secret traefik-tls -o yaml  
apiVersion: v1  
data:  
  tls.crt: someCertificateData  
  tls.key: someCertificateData  
kind: Secret  
metadata:  
  creationTimestamp: "2023-06-27T15:27:12Z"  
  name: traefik-tls  
  namespace: hydrolix  
  resourceVersion: "32029846"  
  uid: 33c06aa2-b81f-44a9-9123-c24e1af94bb5  
type: kubernetes.io/tls

Enable TLS

Once the certificate is deployed, enable TLS for network services by changing the protocol in the hydrolix_url in your Hydrolix spec from http to https:

hydrolix_url: https://${myhost}.hydrolix.live

After changing the protocol, traefik should restart and use the new certificate.

See also Enable TLS.

Troubleshooting

There are some specific troubleshooting steps when using the Hydrolix acme-client and also common Certificate Troubleshooting tips.

The certificate init-acme and acme-renewal jobs both retry upon failure up to six (6) times before stopping. This isn't configurable.

Verify the cluster job runs

To find the init job, run:

$ kubectl -n <namespace> get job --field-selector status.successful=1 | grep acme
init-acme-509c50f0              1/1           21s        24h

This job may disappear if the service has been running for more than a day. Hydrolix clears successful job notifications automatically.

Verify Kubernetes cron job runs

A Kubernetes cron job is also created at startup. To verify, find the acme-renewal job:

$ kubectl -n <namespace> get cronjob
NAME                SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
acme-renewal        0 2 * * SUN   False     0        <none>          24h

Verify startup

List the pods to locate the identifier of the init-acme job.

kubectl -n <namespace> get pods

With the pod identifier, examine the logs:

kubectl -n <namespace> logs init-acme-509c50f0-2zghd

An example of successful initialization

Attempt 1/300: Routing not ready, retrying in 1s...
External routing confirmed working! Ready to proceed with ACME challenge.

Requesting a new certificate
2025/06/03 17:28:47 No key found for account [email protected]. Generating a P256 key.
  [ ... ]

An example of unsuccessful initialization

Testing external routing via ${myhost}.hydrolix.live...
Attempt 1/300: Routing not ready, retrying in 1s...
 [ ... ]
Attempt 300/300: Routing not ready, retrying in 1s...
External routing test failed after 300 attempts.

The above failure indicates that the test client can't reach the configured hostname in the Hydrolix spec hydrolix_url.

How to fix

• examine the hostname for spelling errors, check to see if you can perform a DNS lookup on the name
• attempt to reach the hostname yourself via http://{myhost}.hydrolix.live/

Remove certificate and jobs

To wipe the slate clean and start over.

1. Remove the acme-account and traefik-tls secrets.

   Shell

   kubectl -n <namespace> delete secret traefik-tls
   kubectl -n <namespace> delete secret acme-account
   

2. Remove the init-acme job.

   Shell

   kubectl -n <namespace> delete job init-acme-XYZ  # -- find exact name running in cluster
   

3. Remove the acme-renewal cronjob.

   Shell

   kubectl -n <namespace> delete cronjob acme-renewal
   

📘

Hydrolix-generated certificates update automatically

If you're having issues with renewal, please contact Hydrolix Support.