External Postgres - GKE

Deploying with External Postgres

There are occasions where a more resilient deployment of PostGres is needed or several Hydrolix clusters want to access the same bucket. The following describes how this is achieved within GKE.

🚧

Note

If you are looking to migrate a cluster that already has data loaded please contact [email protected] before commencing this change. The catalog (PostGres) is a critical part of the Hydrolix service and you can cause irreparable damage to the cluster if it is moved incorrectly.

Create an External PostGres instance

Hydrolix suggests using the Google Cloud SQL service to create your PostGres instance. It can be created as a high availability deployments, backups, deletion protection etc all built in.

Hydrolix would recommend sizing your instances based on the sizing criteria specified within Scale Profiles, For example Mega would be 100GB Disk, 6CPU, 24G of Memory.

In addition Hydrolix would recommend deploying it without a public IP. To ensure connectivity with the K8's cluster it should therefore be deployed within the same VPC in the same region (to ensure good performance). More information on how Kubernetes and Cloud SQL can work together can be found [here[(https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#private-ip).

Creating a Hydrolix cluster Yaml.

The hydrolix-cluster command generates the hydrolixcluster.yaml deployment file. We have provided a number of scale profiles for various cloud providers and deployment sizes. You can specify a profile using the scale-profile flag. You can also edit the hydrolixcluster.yaml to tune each deployment to your resource requirements. The following instructions create a dev scale deployment and apply it to your cluster.

hkt hydrolix-cluster --scale-profile dev --ip-allowlist `curl -s ifconfig.me`/32 > hydrolixcluster.yaml

Once you have the basic file created you will need to edit it.

Edit your hydrolixcluster.yaml

Open the hydrolixcluster.yaml in your favourite text editor and you will need to edit the following values::

---
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: <NameSpace>                #<--- Should already be set
spec:
  admin_email: <admin email>       #<--- Should already be set
  db_bucket_url: <bucket path>     #<--- Should already be set
  db_bucket_region: <region>       #<--- Should already be set
  hydrolix_url: <hostname to use>  #<--- Should already be set
  catalog_db_admin_user: postgres  #<--- Add the admin user "postgres" to your config
  catalog_db_admin_db: postgres    #<--- Add the admin db "postgres" to your config
  catalog_db_host: <YOU HOST/IP>   #<--- Add the IP for your cluster
  pg_ssl_mode: <disable/require>   #<--- Set whether you would like to use SSL.
  ip_allowlist:
    - 111.222.333.444/32           #<--- Should be already set
  scale_profile: dev               #<--- Should be already set
  scale:
    postgres:
      replicas: 0                  #<---- Don't forget to set the internal postgres to 0

For example:

---
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: myhdxdeployment
spec:
  admin_email: [email protected]
  db_bucket_url: gs://myhdxdeployment
  db_bucket_region: us-central1
  hydrolix_url: http://my.hdxdeploymente.com
  catalog_db_admin_user: postgres
  catalog_db_admin_db: postgres
  catalog_db_host: 11.22.11.22
  pg_ssl_mode: disable
  ip_allowlist:
    - 111.222.333.444/32
  scale_profile: dev
  scale:
    postgres:
      replicas: 0

❗️

PostGres

In the Scale portion of your file make sure to set PostGres to 0 replicas in the scale section. This is so the postgres instance isn't started.

   postgres:
     replicas: 0

Create your Secret

The PostGres Secret should be held within a curated secret within Kubernetes.

---
apiVersion: v1
kind: Secret
metadata:
  name: curated
  namespace: <namespace>
stringData:
  ROOT_DB_PASSWORD: <the password to your postgres>
type: Opaque

For example:

---
apiVersion: v1
kind: Secret
metadata:
  name: curated
  namespace: myhdxdeployment
stringData:
  ROOT_DB_PASSWORD: mysupersecretpassword
type: Opaque

Apply your configuration

The following commands will apply these settings.

kubectl apply -f secrects.yaml
kubectl apply -f hydrolixcluster.yaml

📘

Already Running Cluster

If you have created this after you have already deployed, you will need to do a kubectl rollout restart deployment for the cluster.