Migration to EKS Cluster

Setup environment variables⚓︎

. We'd recommend setting up environment variables, to make the installation easier:

export HKT_VERSION=v3.19.4

export HDX_DB_BUCKET_URL=s3://hdxcli-flpp5zsi
export HDX_DB_BUCKET=hdxcli-flpp5zsi                                                           
export HDX_BUCKET_REGION=us-east-2                                                                   
export HDX_HYDROLIX_URL=http://my-eks.hydrolix.net                                                       
export HDX_KUBERNETES_NAMESPACE=develeopment-eks                                                             
export HDX_ADMIN_EMAIL=your@email.com 

export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query "Account" --output text)"
export AWS_STORAGE_ROLE="arn:aws:iam::$AWS_ACCOUNT_ID:role/$HDX_KUBERNETES_NAMESPACE-bucket"

Create the bucket policy⚓︎

read -r -d '' POLICY_DOC << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::$HDX_DB_BUCKET",
                "arn:aws:s3:::hdx-public"
            ]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": [
                "arn:aws:s3:::$HDX_DB_BUCKET/*",
                "arn:aws:s3:::hdx-public/*"
            ]
        }
    ]
}
EOF

## Apply the policy

aws iam create-policy --policy-name "$HDX_KUBERNETES_NAMESPACE-bucket" --policy-document "$POLICY_DOC"

Create node configuration and basic cluster eksctl.yaml⚓︎

In order to retain access to the storage bucket and RDS Catalog services we need to create the EKS cluster in the same VPC as the previous deployment.

aws ec2 describe-subnets --filters "Name=tag:aws:cloudformation:stack-name,Values=hdxcli-flpp5zsi-self-deploy" --region us-east-2 --query 'Subnets[*].[AvailabilityZone, SubnetId, Tags[?Key==`Group`].Value]'

cat > eksctl.yaml << EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: $KUBERNETES_CLUSTER 
  region: $HDX_BUCKET_REGION 

addons:
  - name: aws-ebs-csi-driver

iam:
  withOIDC: true

vpc:
  subnets:
    private:
        rds-region-a:
          id: subnet-0bf58ae3fdda44096
        rds-region-b:       
            id: subnet-01c6fa5598707094e   
      private-region-a:
          id: subnet-03626601d35dfcb43
        private-region-b:  
          id: subnet-023cbd238b85a3bd2
      query-region-a:
          id: subnet-070224d5a1e847795
        query-region-b:   
          id: subnet-0bc7a8edd4cb17fe9

managedNodeGroups:
  - name: nodegroup0 
    instanceType: m5n.4xlarge
    minSize: 3
    maxSize: 30
    desiredCapacity: 4
    volumeSize: 256
    privateNetworking: true
EOF

Dev Clusters

If you are looking to create a dev cluster we'd suggest changing the instanceType to t3.2xlarge.

This can take a little while to create the cluster.

eksctl create cluster -f eksctl.yaml

Retrieve the Identity Provider for the cluster⚓︎

OIDC_PROVIDER="$(aws --region "$REGION" eks describe-cluster --name "$HDX_KUBERNETES_NAMESPACE" \
    --query "cluster.identity.oidc.issuer" \
    --output text | sed -e "s/^https:\/\///")"

Check the `OIDC_PROVIDER` Environment Variable⚓︎

echo "$OIDC_PROVIDER"

If this is blank retry the previous step to retrieve the Identity Provider for the cluster

Create an IAM Policy for Kubernetes Service Accounts⚓︎

read -r -d '' SA_POLICY_DOC << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$OIDC_PROVIDER:aud": "sts.amazonaws.com",
          "$OIDC_PROVIDER:sub": "system:serviceaccount:$HDX_KUBERNETES_NAMESPACE:hydrolix"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$OIDC_PROVIDER:aud": "sts.amazonaws.com",
          "$OIDC_PROVIDER:sub": "system:serviceaccount:$HDX_KUBERNETES_NAMESPACE:turbine-api"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$OIDC_PROVIDER:aud": "sts.amazonaws.com",
          "$OIDC_PROVIDER:sub": "system:serviceaccount:$HDX_KUBERNETES_NAMESPACE:vector"
        }
      }
    }
  ]
}
EOF

Create an IAM Role for Kubernetes Service Accounts⚓︎

aws iam create-role --role-name "$HDX_KUBERNETES_NAMESPACE-bucket" \
    --assume-role-policy-document "$SA_POLICY_DOC" \
    --description "$HDX_KUBERNETES_NAMESPACE-bucket"

Attach the Service Account IAM Policy to the Service Account IAM Role⚓︎

aws iam attach-role-policy --role-name "$HDX_KUBERNETES_NAMESPACE-bucket" \
    --policy-arn="arn:aws:iam::$AWS_ACCOUNT_ID:policy/$HDX_KUBERNETES_NAMESPACE-bucket"

## Grab the ARN into a variable
export AWS_STORAGE_ROLE="arn:aws:iam::$AWS_ACCOUNT_ID:role/$HDX_KUBERNETES_NAMESPACE-bucket"

Create the namespace⚓︎

kubectl create namespace $HDX_KUBERNETES_NAMESPACE

For ease of use, set your new namespace as a default:

kubectl config set-context --current --namespace="$HDX_KUBERNETES_NAMESPACE"

Setup GP3 High Performance Disk⚓︎

cat > gp3.yaml << EOF
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF

kubectl apply -f gp3.yaml

Todo: Find CATALOG_DB_HOST: add to curated configmap

kubectl apply -f pg-configmap.yaml

Todo: find ROOT_DB_PASSWORD add to curated secret as base64 encoded

kubectl apply -f pg-secret.yaml

Setup Autoscaling in EKS⚓︎

Deploy to EKS⚓︎

hkt operator-resources > operator.yaml
kubectl apply -f operator.yaml

Deploy Hydrolix Cluster with internal Postgres scaled off

1 2	`hkt hydrolix-cluster --scale-profile dev > hydrolixcluster.yaml kubectl apply -f hydrolixcluster.yaml && kubectl scale --replicas=0 sts/postgres`