Enable TLS
Overview
Hydrolix provides multiple ways to manage certificates on your cluster:
- Generate and manage the certificate entirely within Hydrolix
- Upload your own certificate to your cluster
Use TLS to securely communicate with Hydrolix endpoints. The traefik
component manages TLS. By default, Hydrolix uses a self-signed certificate.
To use cert-manager
to generate and manage certificates, see Add a Custom Certificate .
Enable TLS
To enable TLS, update the protocol section of the hydrolix_url
component to https
in your hydrolixcluster.yaml
configuration file:
hydrolix_url: https://demo.hydrolix.net -> Forces Hydrolix cluster to use TLS
hydrolix_url: http://demo.hydrolix.net -> Forces Hydrolix cluster to use plain HTTP
Native port change from 9000 to 9440
When TLS is enabled, the native ClickHouse interface listens on port 9440.
Hydrolix-managed certificate
Use Hydrolix to generate and manage a TLS certificate for you.
Prerequisites
Before you begin:
- Create an active DNS record for your Hydrolix cluster's URL. This is usually an A record.
- Ensure public access to the IP that hosts your cluster for the certificate challenge
Generate a certificate in Hydrolix
- Set the configuration option
acme_enabled
totrue
inhydrolixcluster.yaml
. - Load the configuration changes to your Hydrolix cluster.
Hydrolix automatically generates a certificate for your cluster and stores it in a Kubernetes secret namedtraefik-tls
.
Generate a certificate with an allowlist
If your cluster uses an allowlist, Hydrolix adds the Buypass ACME certificate provider IP address to the allowlist.
Hydrolix adds the IP automatically to the spec.ip.allowlist
cluster configuration. Check the hydrolixcluster.yaml
file to verify that the correct IP address is present.
Generate a certificate without an allowlist
If your cluster doesn't use an allowlist, Hydrolix uses the Let's Encrypt certificate provider.
Hydrolix refreshes the certificate weekly, and the certificate is valid for 90 days.
Confirm certificate generation
To confirm certificate deployment, run the following command to view details of the certificate.
$ kubectl -n <namespace> get secret traefik-tls -o yaml
For example:
$ kubectl -n hydrolix get secret traefik-tls -o yaml
apiVersion: v1
data:
tls.crt: someCertificateData
tls.key: someCertificateData
kind: Secret
metadata:
creationTimestamp: "2023-06-27T15:27:12Z"
name: traefik-tls
namespace: hydrolix
resourceVersion: "32029846"
uid: 33c06aa2-b81f-44a9-9123-c24e1af94bb5
type: kubernetes.io/tls
Set the acme_enabled
value to true
in the hydrolixcluster.yaml
file, and apply the changes. Hydrolix starts the init
job in the cluster, and creates a cron job called acme-renewal
.
Verify the init
job runs
init
job runsTo find the init job, run:
$ kubectl -n <namespace> get job --field-selector status.successful=1 | grep acme
init-acme-509c50f0 1/1 21s 24h
This job may disappear if the service has been running for more than a day. Hydrolix clears successful job notifications automatically.
Verify Kubernetes cron job runs
A Kubernetes cron job is also created at startup. To verify, find the acme-renewal
job:
$ kubectl -n <namespace> get cronjob
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
acme-renewal 0 2 * * SUN False 0 <none> 24h
Hydrolix-generated certificates update automatically
If you're having issues with renewal, please contact Hydrolix Support.
Use or renew your own certificate
-
To add or renew a certificate:
kubectl create secret tls traefik-tls --key=certificates.key --cert=fullchain.pem
This creates a secret in Kubernetes called
traefik-tls
and stores your certificate and private key in that
secret. Traefik automatically checks for thetraefik-tls
secret, and uses it if valid. -
To apply the changes, restart
traefik
:kubectl rollout restart deployment traefik
Kubernetes certificate chain order
Kubernetes requires full chain certificates. The certificate chain should begin with your certificate, continue with intermediate certificates down the chain, and end with the root certificate:
\-----BEGIN CERTIFICATE-----
{ Your issued Certificate }
-----END CERTIFICATE-----
----BEGIN CERTIFICATE-----
{ Intermediate Certificate }
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
{ Root Certificate }
-----END CERTIFICATE-----
Troubleshoot certificates
The certificate request process retries six times before it stops. To check the status and locate possible problems, look at these logs:
- The
init-acme
job logs - The
acme
pod logs
No cron job listed
If the cron job is not running, the initial init-script may have failed.
How to fix:
-
Delete the
init-acme
job if it exists.kubectl -n <namespace> get jobs | grep init-acme kubectl -n <namespace> delete job <job-name>
-
Delete the acme-renewal cron job if it exists:
kubectl -n <namespace> get cronjobs | grep acme-renewal kubectl -n <namespace> delete cronjob acme-renewal
-
Restart the operator to trigger the init job.
kubectl rollout restart deployment operator -n <namespace>
LetsEncrypt is busy
Check the acme
pod logs for a line similar to this:
acme: error: 0 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Service busy; retry later., url:
This can happen due to the number of requests going to the Let's Encrypt service.
How to fix:
Wait 30 minutes for the request backlog and try again.
AWS-specific allowlist
The VPC security groups for the AWS load balancer have a maximum of 60 rules. If your AWS load balancer hits the limit, Hydrolix can't add the required IP addresses for certificate renewal.
How to fix:
- Temporarily set the allowlist to
0.0.0.0/0
. - Apply the configuration to the
hydrolixcluster.yaml
file.
kubectl apply -f hydrolixcluster.yaml --namespace <namespace>
- Wait for the
init-acme
job to complete.
kubectl -n <namespace> get job --field-selector status.successful=1 | grep acme
- Verify that the job completed successfully, and that a
traefik-tls
secret exists. - Revert the changes made to
hydrolixcluster.yaml
and verify that theacme-renewal
cronjob exists.
If this option isn't feasible, use a custom certificate method instead.
Updated 13 days ago