Skip to content

Hydrolix ACME Client

Overview⚓︎

Hydrolix includes software for automatic certificate acquisition and renewal.

This is the default and recommended option for managing TLS in a Hydrolix cluster. See Enable TLS for more TLS options.

The TLS industry has adopted a standard called Automatic Certificate Management Environment (ACME) which can be supported by a Certification Authority (CA).

This page describes how to use the ACME client in a Hydrolix cluster for automated provisioning of TLS certificates.

The Hydrolix implementation automates sourcing certificates from the Let's Encrypt CA using the Lego project.

Prerequisites⚓︎

Before you can start this guide, you'll need the following:

  • A deployed Hydrolix cluster
  • A publicly reachable IP address on the cluster
  • An active DNS record for the cluster's name

Request a certificate⚓︎

Request a certificate with only exact-match hostnames. Wildcards aren't supported. This uses HTTP-01 validation with the CA.

  1. Confirm hostnames.
  2. Remove existing certificates by dropping the traefik-tls Kubernetes secret.
  3. Prepare for HTTP-01 validation.
  4. Set the tunable acme_enabled to true in your Hydrolix cluster manifest.
  5. Apply the changes to the Hydrolix cluster manifest.
  6. Monitor the acme service.
  7. Verify certificate and reachability.

Request a certificate with a wildcard hostname derived from the cluster name. This uses DNS-01 validation with the CA.

  1. Confirm hostnames.
  2. Remove existing certificates by dropping the traefik-tls Kubernetes secret.
  3. Configure for DNS-01 validation.
  4. Set the tunables acme_enabled and issue_wildcard_cert to true in your Hydrolix cluster manifest.
  5. Apply the changes to the Hydrolix cluster manifest.
  6. Monitor the acme service.
  7. Verify certificate and reachability.

Confirm hostnames⚓︎

These tunables set the TLS hostnames used when requesting a certificate. They become certificate's Subject Alternative Names (SANs).

  1. Confirm spelling of hydrolix_url. This is the primary name on the certificate request.
  2. Set issue_wildcard_cert to include a wildcard TLS hostname derived from the hydrolix_url.
  3. Confirm spelling of all alt_names, if you are using them. These are additional names to be included in the certificate request.

Remove existing certificates⚓︎

Removing certificates causes TLS termination to break

The Traefik reverse proxy loads a self-signed default certificate when the traefik-tls secret is absent. Clients won't be able to validate this default certificate.

New connections fail until a certificate is successfully provisioned or installed. Existing connections remain open.

  1. Remove the traefik-tls secret. When absent, the acme service requests a certificate.

    1
    kubectl delete secret traefik-tls

  2. (optional) Remove the acme-account secret. When absent, the acme service creates a new account with the CA using the email address in admin_email.

    1
    kubectl delete secret acme-account

Configure DNS-01 validation⚓︎

ACME specification requires DNS-01 to validate wildcard hostnames. Hydrolix supports only AWS Route53 as an acme_dns_provider for installing the challenge tokens.

  1. Create an IAM policy granting minimal permissions. Replace {hosted-zone-id} and _acme-challenge.hostname.hydrolix.live with the Route53 hosted zone ID and the Hydrolix cluster hostname in this example IAM policy.

    Example IAM policy 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/{hosted-zone-id}"                // (2)!
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/{hosted-zone-id}"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "route53:ChangeResourceRecordSetsNormalizedRecordNames": [
                        "_acme-challenge.hostname.hydrolix.live"               // (1)!
                    ],
                    "route53:ChangeResourceRecordSetsRecordTypes": [
                        "TXT"
                    ]
                }
            }
        }
    ]
}
    1. This is the fully qualified domain name (FQDN) the CA looks up when using DNS to validate a wildcard TLS hostname. The Hydrolix ACME client manipulates the TXT DNS record _acme-challenge.hostname.hydrolix.live to prove control over the TLS hostname *.hostname.hydrolix.live.
    2. Use the hosted zone ID of the Route53 zone holding the _acme-challenge FQDN.

  2. Attach the policy to a user by adding IAM identity permissions. This can be done directly or by attaching to an identity (user group, role) that includes the user.

  3. Use the AWS Guide to create a new IAM access key and secret.

  4. In hydrolixcluster.yaml configuration, set issue_wildcard_cert: true. This switches the acme service to use DNS-01 validation.

  5. Create additional entries in the curated secret holding the access key ID and secret access key generated for the AWS user. Set the AWS region and hosted zone ID corresponding to your Route53 configuration.

    1
2
3
4
    ROUTE53_AWS_ACCESS_KEY_ID = {Access key ID}
ROUTE53_AWS_SECRET_ACCESS_KEY = {Secret access key}
ROUTE53_AWS_HOSTED_ZONE_ID = {Route53 zone ID}
ROUTE53_AWS_REGION = {Route53 region}

Monitor the acme service⚓︎

  1. Watch the acme logs. Use any Kubernetes log tooling.

    1
    kubectl logs -f -l app=acme

  2. (optional) Restart the acme service.

    1
    kubectl rollout restart deployment acme

See also Logging examples.

Verify certificate⚓︎

To verify successful storage and server installation of the certificate, use the following commands.

1
2
3
4
5
6
7
$ curl \
  --fail \
  --silent \
  --output /dev/null \
  --write-out '%{stderr}response %{response_code} url %{url} exit %{exitcode}\n' \
  --  https://hostname.hydrolix.live/
response 200 url https://hostname.hydrolix.live/ exit 0

This is a successful connection. TLS failures generate non-zero exits.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ HOSTNAME=hostname.hydrolix.live
$ echo Q \
  | openssl 2>/dev/null s_client -connect "${HOSTNAME}:443" -servername "${HOSTNAME}" \
  | openssl x509 -text -noout \
  | grep -A2 -E 'X509v3 Subject Alternative Name|Validity'
        Validity
            Not Before: Aug 19 18:50:53 2026 GMT
            Not After : Nov 17 18:50:52 2026 GMT
--
            X509v3 Subject Alternative Name:
                DNS:*.hostname.hydrolix.live, DNS:hostname.hydrolix.live
            X509v3 Certificate Policies:

All certificate hostnames are listed in the Subject Alternative Names field. Not After is the expiration date.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ kubectl get secret traefik-tls -o yaml
apiVersion: v1
data:
  tls.crt: LS0tLS1CRUdJTiBDRVooooLONGiBASE64iSTRINGiSECRETooooolDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0ooooLONGiBASE64iSTRINGiSECREToooootFWS0tLS0tCg==
kind: Secret
metadata:
  annotations:
    acme.cert/challenge-type: dns-01
    acme.cert/domains: '[hostname.hydrolix.live *.hostname.hydrolix.live]'
    acme.cert/renewed-at: "2026-08-19T19:49:24Z"
  creationTimestamp: "2026-08-19T19:46:13Z"
  name: traefik-tls
  namespace: namespace-name
  resourceVersion: "255447695"
  uid: 9c102b9d-b362-460a-976a-f18a2c634a8d
type: kubernetes.io/tls

Configuration⚓︎

Hydrolix uses Let's Encrypt as the preferred CA. Use a different CA by setting acme_provider tunable to the provider's URL.

The Hydrolix ACME client supports HTTP-01 and DNS-01. The ACME specification requires DNS-01 validation for wildcard hostnames.

Certificate validity period depends on the CA. The service checks regularly for upcoming expiration and renews early.

Certificate key type⚓︎

Hydrolix version 6.3 and later supports elliptic-curve keys. Set traefik_tls_key_type to choose the private key type the CA issues. The default is RSA2048.

Value Key
EC256 Elliptic curve, P-256
EC384 Elliptic curve, P-384
RSA2048 RSA, 2048-bit (default)
RSA3072 RSA, 3072-bit
RSA4096 RSA, 4096-bit
RSA8192 RSA, 8192-bit

The operator rejects any other value.

HTTP-01 validation⚓︎

By default, HTTP-01 validation is used. All hostnames must be exact match, no wildcards.

The acme service retrieves the challenge token from the CA and serves the token from URL https://hostname.hydrolix.live/.well-known/acme-challenge/<TOKEN>.

Using HTTP-01 validation isn't compatible with a strict ip_allowlist

The CA must be able to reach the token challenge URL to validate control of the domain.

When ip_allowlist is used, the external load balancer only allows incoming connections from IPs matching the list. This can block the CA's validation requests. Avoid this blockage by installing permissive 0.0.0.0/0 in the external load balancer ip_allowlist.

To enforce strict IP controls, use the in-cluster Traefik IP Allowlist instead of the external load balancer ip_allowlist.

This cluster-level IP access control list always allows the ACME HTTP-01 challenge requests.

DNS-01 validation⚓︎

When issue_wildcard_cert=true, a wildcard hostname is formed from the hostname in the hydrolix_url and DNS-01 validation is used.

The acme service installs the challenge tokens using API access to the acme_dns_provider. The only valid provider is route53.

ACME client process⚓︎

The tunable acme_enabled signals the operator to start the acme service.

Hostname collection⚓︎

The acme software collects hostnames from these configuration settings to include in the certificate request.

Runtime behavior⚓︎

The process in the acme pod initiates a certificate request

  • if there's no current certificate in traefik-tls secret
  • if the current certificate doesn't cover all the collected names
  • if the current certificate expires sooner than acme_renew_certs_with_expiry_days days, which defaults to 30

The acme service interacts with the acme_provider to request a certificate, install the challenge for validation, collect the issued certificate, and store it in the traefik-tls Kubernetes secret.

Upon failure, the acme process attempts again after 15 minutes. A higher frequency of attempts risks exceeding the CA's validation limits, which apply per-hour, per-hostname.

Expiration and monitoring⚓︎

The acme service checks remaining certificate validity every acme_cert_check_interval_in_minutes and initiates a certificate request when the certificate expires sooner than acme_renew_certs_with_expiry_days.

The cert-expiry-check daily cronjob updates the number of seconds the traefik-tls certificate remains valid as metric fqdn_cert_expiry.

Hydrolix doesn't provide any email notification of certificate expiration.

Troubleshoot⚓︎

There are some specific troubleshooting steps when using the Hydrolix acme-client and also common Certificate Troubleshooting tips.

  1. Check the hostname for spelling errors and correct any mistakes.
  2. Verify a successful DNS lookup to one or more DNS resolvers for the hostname.
  3. Confirm from your workstation with the host $HDX_HOSTNAME command; if it fails, confirm that the authoritative DNS entry is present.
  4. Confirm using a public resolver with the command host $HDX_HOSTNAME 8.8.8.8. If it fails, diagnose and resolve the authoritative DNS issue.
  5. Confirm the acme pod is running, see Monitor the acme service.
  6. If using HTTP-01 validation, see Enforce IP access controls.

Enforce IP access controls⚓︎

See Configure IP Access to adjust the IP access control list installed in the external load balancer. HTTP-01 validation isn't compatible with a strict ip_allowlist.

See Traefik IP Allowlist to configure the IP access control list installed in the Traefik reverse proxy. HTTP-01 requests are always allowed.

Logging examples⚓︎

When no certificate exists in traefik-tls, the service begins the certificate request process. In this example, issue_wildcard_cert was true and the DNS-01 challenge is used.

ACME Pod Startup with No Certificate, Acquire New Certificate 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
time=2026-08-19T15:32:33.008Z level=INFO msg="Starting ACME Certificate Manager..."
time=2026-08-19T15:32:33.009Z level=INFO msg=Configuration:
time=2026-08-19T15:32:33.009Z level=INFO msg="  Email: username@example.com"
time=2026-08-19T15:32:33.009Z level=INFO msg="  Domains: [hostname.hydrolix.live *.hostname.hydrolix.live]"
time=2026-08-19T15:32:33.009Z level=INFO msg="  Challenge Type: dns-01"
time=2026-08-19T15:32:33.009Z level=INFO msg="  CA URL: https://acme-v02.api.letsencrypt.org/directory"
time=2026-08-19T15:32:33.009Z level=INFO msg="[startHealthServer] Health check server listening on port 8081"
time=2026-08-19T15:32:33.009Z level=INFO msg="[cert.ensureCertificate] - Checking certificate status..."
time=2026-08-19T15:32:33.023Z level=WARN msg="Failed to get certificate secret: secrets \"traefik-tls\" not found"
time=2026-08-19T15:32:33.023Z level=INFO msg="[cert.ensureCertificate] - Certificate needs renewal or doesn't exist"
time=2026-08-19T15:32:33.023Z level=INFO msg="Loading ACME account..."
time=2026-08-19T15:32:33.026Z level=INFO msg="No existing ACME account found, creating new one"
time=2026-08-19T15:32:33.337Z level=INFO msg="Registering new ACME account: username@example.com"
2026/08/19 15:32:33 [INFO] acme: Registering account for username@example.com"
time=2026-08-19T15:32:33.487Z level=INFO msg="Successfully registered account: https://acme-v02.api.letsencrypt.org/acme/acct/6666666666"
time=2026-08-19T15:32:33.487Z level=INFO msg="Saving ACME account to secret..."
time=2026-08-19T15:32:33.497Z level=INFO msg="Created new account secret: acme-account"
time=2026-08-19T15:32:33.580Z level=INFO msg="Setting up challenge provider type: dns-01"
time=2026-08-19T15:32:33.580Z level=INFO msg="Setting up DNS-01 challenge with provider: route53"
time=2026-08-19T15:32:33.580Z level=INFO msg="DNS-01 challenge configured successfully"
time=2026-08-19T15:32:33.580Z level=INFO msg="Obtaining certificate from domains: [hostname.hydrolix.live *.hostname.hydrolix.live]"
time=2026-08-19T15:32:33.580Z level=INFO msg="Using challenge type: dns-01"
2026/08/19 15:32:33 [INFO] [hostname.hydrolix.live, *.hostname.hydrolix.live] acme: Obtaining bundled SAN certificate
2026/08/19 15:32:33 [INFO] [*.hostname.hydrolix.live] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/6666666666/777777777777
2026/08/19 15:32:33 [INFO] [hostname.hydrolix.live] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/6666666666/777777777777
2026/08/19 15:32:33 [INFO] [*.hostname.hydrolix.live] acme: use dns-01 solver

     [ ... omitted ... ]

2026/08/19 15:32:33 [INFO] [hostname.hydrolix.live] acme: use dns-01 solver
2026/08/19 15:32:33 [INFO] [*.hostname.hydrolix.live] acme: Preparing to solve DNS-01
2026/08/19 15:32:59 [INFO] [hostname.hydrolix.live] acme: Preparing to solve DNS-01
2026/08/19 15:33:24 [INFO] [*.hostname.hydrolix.live] acme: Trying to solve DNS-01
2026/08/19 15:33:24 [INFO] [*.hostname.hydrolix.live] acme: Checking DNS record propagation. [nameservers=10.128.0.10:53]

     [ ... omitted ... ]

2026/08/19 15:33:44 [INFO] [*.hostname.hydrolix.live] acme: Cleaning DNS-01 challenge
2026/08/19 15:34:09 [INFO] [hostname.hydrolix.live] acme: Cleaning DNS-01 challenge
2026/08/19 15:34:34 [INFO] [hostname.hydrolix.live, *.hostname.hydrolix.live] acme: Validations succeeded; requesting certificates
2026/08/19 15:34:36 [INFO] [hostname.hydrolix.live] Server responded with a certificate.
time=2026-08-19T15:34:36.052Z level=INFO msg="Successfully obtained certificate"
time=2026-08-19T15:34:36.070Z level=INFO msg="Created new certificate secret traefik-tls"

When a certificate already exists in traefik-tls and covers all of the names collected for the cluster, the service sleeps. It doesn't issue a new certificate request.

ACME Pod Startup with Existing Certificate, No Action 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
time=2026-08-19T15:39:09.009Z level=INFO msg="Starting ACME Certificate Manager..."
time=2026-08-19T15:39:09.009Z level=INFO msg=Configuration:
time=2026-08-19T15:39:09.009Z level=INFO msg="  Email: username@example.com"
time=2026-08-19T15:39:09.009Z level=INFO msg="  Domains: [hostname.hydrolix.live *.hostname.hydrolix.live]"
time=2026-08-19T15:39:09.009Z level=INFO msg="  Challenge Type: dns-01"
time=2026-08-19T15:39:09.009Z level=INFO msg="  CA URL: https://acme-v02.api.letsencrypt.org/directory"
time=2026-08-19T15:39:09.009Z level=INFO msg="[cert.ensureCertificate] - Checking certificate status..."
time=2026-08-19T15:39:09.009Z level=INFO msg="[startHealthServer] Health check server listening on port 8081"
time=2026-08-19T15:39:09.026Z level=INFO msg="Domains supported by the certificate" domains="[*.hostname.hydrolix.live hostname.hydrolix.live]"
time=2026-08-19T15:39:09.026Z level=INFO msg="Certificate expires in 89.956180 days with expiry check 30.000000 days"
time=2026-08-19T15:39:09.026Z level=INFO msg="[cert.ensureCertificate] - Certificate is valid, no renewal needed"

The acme client verifies HTTP reachability of the challenge endpoint by initiating a request from an external IP, to simulate an arbitrary source IP. It doesn't begin the certificate request process unless the HTTP-01 ACME challenge test succeeds.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
time=2026-08-19T19:37:45.132Z level=INFO msg="Checking if challenge endpoint is accessible via internet before requesting the certificate"
time=2026-08-19T19:37:55.142Z level=INFO msg="[challenge_check.getURLDataWithRetries] - Request URL: http://hostname.hydrolix.live/.well-known/acme-challenge/test"
time=2026-08-19T19:37:55.142Z level=INFO msg="[challenge_check.getURLDataWithRetries] - Response is empty"
time=2026-08-19T19:37:55.142Z level=ERROR msg="[challenge_check.getURLDataWithRetries] - StatusCode: -100, Request error: Get \"http://hostname.hydrolix.live/.well-known/acme-challenge/test\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time=2026-08-19T19:37:55.142Z level=INFO msg="[challenge_check.getURLDataWithRetries] - Retrying with backoff 1s"

     [ ... omitted ... ]

time=2026-08-19T19:40:24.866Z level=INFO msg="DNS hostname.hydrolix.live resolvable"
time=2026-08-19T19:40:24.866Z level=INFO msg="✗ FAILED: Challenge endpoint is not accessible"
time=2026-08-19T19:40:24.866Z level=ERROR msg="  HTTP Error: Get \"http://hostname.hydrolix.live/.well-known/acme-challenge/test\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time=2026-08-19T19:40:24.866Z level=INFO msg="  Expected URL: http://hostname.hydrolix.live/.well-known/acme-challenge/test"
time=2026-08-19T19:40:24.866Z level=INFO msg=Troubleshooting:
time=2026-08-19T19:40:24.866Z level=INFO msg="  - Ensure LoadBalancer is forwarding port 80"
time=2026-08-19T19:40:24.866Z level=INFO msg="  - Verify pod is listening on port 8080"
time=2026-08-19T19:40:24.866Z level=INFO msg="  - Check firewall/security groups"

When acme discovers a name during Hostname collection not present on the certificate, it begins a new certificate request.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
time=2026-08-19T19:49:22.524Z level=WARN msg="domain *.hostname.hydrolix.live not found in certificate"
time=2026-08-19T19:49:22.524Z level=INFO msg="[cert.ensureCertificate] - Certificate needs renewal or doesn't exist"

     [ ... omitted ... ]

time=2026-08-19T19:49:22.752Z level=INFO msg="Setting up DNS-01 challenge with provider: route53"
time=2026-08-19T19:49:22.753Z level=INFO msg="DNS-01 challenge configured successfully"
time=2026-08-19T19:49:22.753Z level=INFO msg="Obtaining certificate from domains: [hostname.hydrolix.live *.hostname.hydrolix.live]"
time=2026-08-19T19:49:22.753Z level=INFO msg="Using challenge type: dns-01"

     [ ... omitted ... ]

time=2026-08-19T19:49:24.188Z level=INFO msg="Successfully obtained certificate"
time=2026-08-19T19:49:24.195Z level=INFO msg="Updated certificate secret traefik-tls"