AWS EMR Deployment

Overview

AWS EMR (previously Elastic MapReduce) is a cloud-based platform for processing and analyzing large datasets using open-source tools like Apache Spark, Hadoop, and Hive. It automates the provisioning, scaling, and management of compute clusters, making big data processing faster and more cost-effective.

Requirements

Verify the following dependencies, permissions, and runtime requirements for successful setup.

Dependencies

• Hydrolix cluster: A running Hydrolix cluster with a version compatible with your connector version. See the compatibility matrix. Find deployment instructions for your preferred cloud vendor (AWS, Google, Linode, or Microsoft) in Welcome to Hydrolix.
• An AWS account: Used for the EMR cluster.
• An AWS VPC and subnet. The EMR cluster requires access to external resources. It needs to communicate with the Hydrolix cluster's API and one or more S3 buckets. Set up VPCs, subnets, NACLs, and route tables in AWS to support the EMR cluster.

Hydrolix credentials

Setup requires credentials for a Hydrolix account that has the required permissions. These credentials may be either:

• A username and password for a Hydrolix user account
• A service account token. This requires connector v3.4.0 or later. See Service account token setup.

Service account token setup

Run the following commands in one terminal session. They require curl, jq, and a super admin account to perform the API calls. Replace the cluster URL and the admin credentials.

1. Log in and get the org UUID:

   HDX="https://{myhost}.hydrolix.live/config/v1"
   
   LOGIN=$(curl -s -X POST "$HDX/login" \
     -H 'Content-Type: application/json' \
     -d '{"username":"{admin email}","password":"{admin password}"}')
   
   ADMIN_TOKEN=$(echo "$LOGIN" | jq -r '.auth_token.access_token')
   ORG=$(echo "$LOGIN" | jq -r '.orgs[0].uuid')
   

2. Create a role with the permissions listed in Required user permissions:

   ROLE=$(curl -s -X POST "$HDX/roles/" \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{
       "name": "spark-connector-reader",
       "policies": [
         {
           "permissions": ["config_blob_org","catalog_urls_table",
             "view_hdxstorage","view_table","view_transform","view_view",
             "select_sql","select_catalog_sql","select_metadata_sql",
             "show_columns_sql","show_tables_sql","show_databases_sql",
             "dictGet_sql","view_function"],
           "scope_type": "org", "scope_id": "'"$ORG"'"
         },
         { "permissions": ["view_org"] }
       ]
     }')
   
   ROLE_ID=$(echo "$ROLE" | jq -r '.id')
   

3. Create a service account and assign the role to it:

   SA=$(curl -s -X POST "$HDX/service_accounts/" \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"name":"spark-connector"}')
   
   SA_UUID=$(echo "$SA" | jq -r '.uuid')
   
   curl -s -X POST "$HDX/roles/$ROLE_ID/add_user/" \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"users":[{"uuid":"'"$SA_UUID"'"}]}'
   

4. Issue the token:

   curl -s -X POST "$HDX/service_accounts/$SA_UUID/tokens/" \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H 'Content-Type: application/json' -d '{}' | jq -r '.token'
   

The command prints the service account token. Store it in your platform's secrets manager and set it as the service_account_token connector property. Note the token's expiry for rotation.

For more detail on service accounts, roles, and tokens, see Manage Service Accounts.

Required user permissions

The connector requires specific access to query a Hydrolix cluster. Assign the following permissions to the account whose credentials you supply, whether a service account or a user account. Super admin access isn't required.

Permission name	Level
view_org	Global
config_blob_org	Org
view_hdxstorage	Org
catalog_urls_table	Org, project, or table
view_table, view_view, view_transform	Org, project, or table
select_sql, select_catalog_sql, select_metadata_sql	Org, project, or table
show_databases_sql, show_tables_sql, show_columns_sql	Org
dictGet_sql, view_function	Org

To restrict the account to specific data, scope the project-or-table-level permissions to those projects or tables instead of the org. To query multiple tables in the same Hydrolix project, scope those permissions to the project level instead of granting the permissions for each table.

Grant view_org at the global level

Org-level view_org isn't sufficient: the connector discovers the org by listing /orgs, which requires view_org at the global level. Without it, queries fail with the error config not loaded, yet. The Hydrolix UI can't create a global-level policy, so create the role with the Config API using a policy that has no scope fields:

{
    "name": "spark-connector-reader",
    "policies": [
        {
            "permissions": ["config_blob_org","catalog_urls_table",
            "view_hdxstorage","view_table","view_transform","view_view",
            "select_sql","select_catalog_sql","select_metadata_sql",
            "show_columns_sql","show_tables_sql","show_databases_sql",
            "dictGet_sql","view_function"],
            "scope_type": "org",
            "scope_id": "{org uuid}"
        },
        { "permissions": ["view_org"] }
    ]
}

The API stores the second policy, which has no scope fields, as a global-level policy.

Runtime requirements

• EMR version: 7.x

Set up AWS EMR

1. Configure the AWS EMR cluster
2. Configure the Spark Connector and create the cluster
3. (Optional) Create EMR Studio: Each Studio is a self-contained, web-based integrated development environment for Jupyter notebooks that run on AWS EMR clusters. You can read more about using this tool to run workloads on your EMR clusters at Amazon EMR Studio.

Configure the AWS EMR cluster

1. Log into AWS Console and choose the preferred region. For example: us-east-1, us-west-2.
2. Navigate to Amazon EMR.
3. In the left nav, find EMR on EC2.
4. Select Clusters, then click Create cluster.

5. Set the cluster's name then pick the latest EMR 7.8.x version and Spark Interactive application bundle. This automatically selects:

   • Hadoop
   • Spark
   • Hive
   • Livy
   • Jupyter

   

6. In Cluster configuration, choose instance types for each of the three groups of instances. EBS isn't required. Minimum configuration includes the following instance counts:

   • 1 Primary
   • 1 Core
   • 1 Task

7. In the Networking block, select a VPC and subnet. Ensure the VPC and subnet will allow access to the Hydrolix cluster's API and any necessary S3 buckets. Allow AWS to create ElasticMapReduce-Primary and ElasticMapReduce-Core security groups.

SSH and Spark UI access require additional permissions

You may need to modify the generated Security groups if you need SSH or Spark UI access.

Configure the Spark Connector and create the cluster

1. In the Software settings block, enter the following configuration for the EMR cluster corresponding to the EMR version:

   [{
       "Classification": "spark-defaults",
       "Properties": {
         "spark.jars": "{/path/to/spark-connector.jar}",
         "spark.pyspark.python": "/usr/bin/python3",
         "spark.pyspark.virtualenv.bin.path": "/usr/bin/virtualenv",
         "spark.pyspark.virtualenv.enabled": "true",
         "spark.pyspark.virtualenv.type": "native",
         "spark.sql.catalog.hydrolix": "io.hydrolix.connectors.spark.SparkTableCatalog",
         "spark.sql.catalog.hydrolix.cluster_url": "https://hostname.hydrolix.live",
         "spark.sql.extensions": "io.hydrolix.connectors.spark.SummaryUdfExtension"
       }
     },
     {
       "Classification": "livy-conf",
       "Properties": {
         "livy.rsc.jars": "/lib/livy/rsc-jars/*,{/path/to/spark-connector.jar}"
       }
     }
   ]
   

2. Replace {/path/to/spark-connector.jar} with the URL under Spark Connector Version in the Hydrolix Connector for Apache Spark table that's compatible with the Hydrolix cluster version. Replace hostname.hydrolix.live with the Hydrolix cluster URL.

3. Note that {/path/to/spark-connector.jar} is referenced twice: in spark-defaults and livy-conf.

   • The spark-defaults block affects running Spark jobs, spark-shell, or pySpark over SSH.
   • The livy-conf block impacts running Spark jobs through an AWS Notebook.

4. In Security configuration and EC2 key pair you can configure cluster access through SSH. Follow the instructions in Create a security configuration with the Amazon EMR console or with the AWS CLI to create a security configuration for the EMR cluster. To set up SSH access to the EMR cluster nodes, see Use an EC2 key pair for SSH credentials for Amazon EMR .

5. In the Identity and Access Management (IAM) roles block, click Create a service role > Create an instance profile. For credential handling (or any other custom permissions), you can attach additional policies to the generated EMR service role in IAM at a later time.

1. Click Create cluster.

(Optional) Create EMR Studio

Follow the instructions in Create an EMR Studio to create an EMR Studio instance. Note the following requirements to ensure EMR Studio works with your previously-created EMR Cluster:

1. While creating the EMR Studio instance: In the Networking and security block, set the same VPC and subnet as your EMR cluster and choose Default security group. After doing so, you can Create Studio.
2. After creating the EMR Studio instance: In the Workspaces (notebooks) submenu, select your workspace then Attach cluster. Choose your EMR cluster and security groups should appear automatically. Press Attach cluster and launch.

After creation, the Workspace is started automatically (Status: Ready). When trying to attach a cluster, it may not show any available clusters. In that case, perform Actions > Stop on the workspace. At this point, Attach cluster should show your newly-created EMR cluster.

If Jupyter notebook fails, check browser settings

After starting the EMR Studio instance, a Jupyter notebook should automatically start in a new tab. If the notebook doesn't start, check your browser settings as it may block the pop-up.

You should now be able to use Spark or PySpark kernels to run jobs on EMR Spark.

Verification

Run the following command to verify that you can successfully log in to your Hydrolix cluster:

spark.sql("use hydrolix").show()

Secrets management

There are multiple options for credentials management in AWS. This example uses the AWS Systems Manager Parameter Store. There are other methods of storing and retrieving secrets in the AWS Secrets Manager User Guide.

Create credentials

To store your Hydrolix credentials as parameters using the AWS Console, AWS CLI, or Tools for Windows PowerShell, see Creating Parameter Store parameters in Systems Manager.

Set PySpark notebook credentials

Open EMR Studio and configure virtualenv

Open the EMR Studio attached to your EMR cluster and use the PySpark notebook kernel to install the boto3 library using venv:

%%configure -f
{
  "conf": {
    "spark.pyspark.virtualenv.enabled": "true",
    "spark.pyspark.virtualenv.bin.path": "/usr/bin/virtualenv",
    "spark.pyspark.virtualenv.type": "native",
    "spark.pyspark.python": "/usr/bin/python3"
  }

Install boto3

Install the boto3 package:

sc.install_pypi_package("boto3")

This creates a Spark session.

Authentication mode

The connector authenticates to the Hydrolix cluster with one of two kinds of Hydrolix credentials:

• Hydrolix user account: Set username and password to the Hydrolix user account credentials.

• Service account token: Set the service_account_token property. username and password are ignored.

  For example:

  spark.sql.catalog.hydrolix io.hydrolix.connectors.spark.SparkTableCatalog
  spark.sql.extensions io.hydrolix.connectors.spark.SummaryUdfExtension
  spark.sql.catalog.hydrolix.cluster_url https://hostname.hydrolix.live
  spark.sql.catalog.hydrolix.service_account_token {token}
  

  This example uses hydrolix as the catalog name; use whatever name you configured for the connector. On Unity Catalog-enabled Databricks workspaces, the catalog name must be spark_catalog.

Store credentials in your platform's secrets manager rather than pasting them into the cluster configuration.

Service account tokens require connector v3.4.0 or later

Connector versions before v3.4.0 (up to and including v3.0.0-v5.6.0) must authenticate to the cluster's Config API with username and password. Service account tokens won't work.

Retrieve and set credentials

To get the credentials and set them in a Spark context, use this Python snippet:

import boto3
ssm = boto3.client("ssm", region_name="{region}")
username = ssm.get_parameter(Name="{/path/to/username}", WithDecryption=True)["Parameter"]["Value"]
password = ssm.get_parameter(Name="{/path/to/password}", WithDecryption=True)["Parameter"]["Value"]
spark.conf.set("spark.sql.catalog.hydrolix.username", username)
spark.conf.set("spark.sql.catalog.hydrolix.password", password)

Alternatively, authenticate with a service account token (connector v3.4.0 or later) instead of username/password:

import boto3
ssm = boto3.client("ssm", region_name="{region}")
token = ssm.get_parameter(Name="{/path/to/token}", WithDecryption=True)["Parameter"]["Value"]
spark.conf.set("spark.sql.catalog.hydrolix.service_account_token", token)

Create an EMR Serverless cluster

Using EMR Serverless requires EMR Studio. See (Optional) Create EMR Studio for instructions on setting up EMR Studio.

1. Log in to the AWS Console and select the appropriate region (for example, us-east-1).
2. Open Amazon EMR > EMR Serverless.
3. Select Manage applications > Create application.
4. Set the application name. For example, HDX-Spark-Connector.
5. Under Application setup options select Use custom settings.

6. In the Pre-initialized capacity section, set the driver and executors configuration:

   • number
   • vCPUs
   • memory
   • disk size

7. Under interactive endpoints, select both options:

   • Enable endpoint for EMR studio
   • Enable Apache Livy endpoint

8. (Optional) Under Network connections, set your VPC and Subnet if you plan on using EMR Serverless with other AWS services such as SageMaker Studio.

9. In the Application configuration, set the following Hydrolix configuration:

   {
     "runtimeConfiguration": [
       {
         "classification": "spark-defaults",
         "properties": {
           "spark.jars": "{spark-connector-location}",
           "spark.sql.extensions": "io.hydrolix.connectors.spark.SummaryUdfExtension",
           "spark.sql.catalog.hydrolix": "io.hydrolix.connectors.spark.SparkTableCatalog",
           "spark.sql.catalog.hydrolix.cluster_url": "https://hostname.hydrolix.live"
         }
       }
     ]
   }
   

10. Choose Create application.

EMR Studio Runtime Role

A runtime role isn't necessary to create a studio or cluster.

A runtime role, however, is necessary to attach EMR Studio or SageMaker Studio notebooks to an EMR Serverless cluster. Configure it using the EMR serverless cluster setup wizard with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ServerlessTrustPolicy",
      "Effect": "Allow",
      "Principal": {
        "Service": "emr-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Additional policies can be added to access S3 buckets, AWS Glue, and more.

Integrate AWS Sagemaker

Once AWS EMR and Hydrolix are working together, they can be used as a data source and execution engine for training and publishing AI models from AWS Sagemaker.

AWS infrastructure also allows multiple ways for Sagemaker notebooks to control EMR instances. In all cases, you must connect Sagemaker to the EMR cluster.

To integrate SageMaker Studio with EMR Serverless, complete the EMR Studio and EMR Serverless setup steps first.

SageMaker AI Notebook and EMR

One method of integrating AWS Sagemaker is to configure a Sagemaker AI Notebook pointed to a running EMR instance. You can do so using the instructions in Control an Amazon EMR Spark Instance Using a Notebook.

Ensure network settings allow Sagemaker access to the EMR cluster

Ensure Sagemaker and the EMR cluster are configured within the same VPC and Subnet. Don't disable direct internet access.

Once integrated, in the notebook and using the PySpark Kernel you may import AI model libraries, train the models on data received from querying Hydrolix, and publish these models to s3 for deployment with Sagemaker.

SageMaker Studio and EMR Serverless

SageMaker Studio is a more complex product that also allows integration with EMR Serverless cluster. You can find instructions for integrating Sagemaker Studio with a serverless cluster at Prepare data using EMR Serverless.

troubleshoot

• Verify the VPC, Subnet, and security group configurations allow EMR Studio access to EMR Serverless.
• The Livy endpoint must be enabled on the EMR Serverless cluster. The SageMaker execution role must allow AccessLivyEndpoints and network configuration must not prevent it from accessing the Livy port. See the steps in EMR Serverless for instructions on enabling the Livy endpoint.
• A single account setup requires the iam:PassRole permission be added to the SageMaker Execution role.

Related information

• Build Better Models with AWS EMR and Hydrolix - How to use the Hydrolix Spark Connector with AWS EMR to run large-scale Spark jobs against Hydrolix data