Batch ingestion loads data from a storage bucket into a target table. We provide the following mechanisms:

• Batch Job API. A one-off task that will load one or more files based on the job configuration and then stop.
• Batch auto-ingest. A continuous task of ingesting new files arriving in a storage bucket, through a combination of table settings, cloud provider pub/sub and cloud storage notification mechanism.

Batch ingest supports CSV and JSON data formats. Hydrolix requires read permissions to access external storage buckets.

The majority of the work is done by the batch-peer and batch-indexer containers in a pod called batch-peer. By default, batch ingest processes one file at a time, with the replicas tunable set to a value of 1. Increase the value of the replicas tunable to increase parallelism. The containers use a predefined resource profile, but you can override memory or storage settings if needed. For more details, see Scale your Kubernetes Cluster.

As of Hydrolix v4.14, if you have summary tables using data from the tables in which you load data, those summary tables will also be populated.

You can configure cloud storage to notify Hydrolix when new data is available for ingest. For more information, see the following pages:

• AWS S3 Notify
• GCP Storage Notify

Create a Batch Ingest Job via the API

👍

Prerequisite Steps

• Create a Project providing a name and description
• Create a Table providing a name and ingest settings
• If the storage bucket from which you're reading requires authentication, create a Credential providing access to a cloud storage bucket containing those source files.
• Create a Transform (Write Schema)

Once you've completed the prerequisites, Create a Batch Job. You must specify the following:

• the table where Hydrolix will store the data
• the transform Hydrolix should use to process the data
• the source where Hydrolix should fetch the data, including the bucket credential ID if needed
• the regex filter Hydrolix should use to limit ingestion to a subset of data (optional)

Note that the batch job format was updated in Hydrolix version 4.19.1. Versions of Hydrolix since v4.19.1 will not accept the old format.

AWS Example

{
    "type": "batch_import",
    "name": "job_sample_data",
    "description": "sample data on aws",
    "settings": {
        "max_active_partitions": 576,
        "max_rows_per_partition": 33554432,
        "max_minutes_per_partition": 20,
        "source": {
            "settings": {
                "bucket_name": "mydatatoingest",  
                "bucket_path": "/mypath",  
                "region": "us-east-1",  
                "cloud": "aws",  
                "credential_id": "11111111-1111-1111-1111-111111111111"
            },
            "table": "sample.data",
            "type": "batch",
            "transform": "mytransform"
        },
        "regex_filter": "^s3://mydatatoingest/mypath/.*.gz"
    }
}

Note that you can omit the credential ID above if the bucket you're using doesn't require credentials.

GCP Example

🚧

GCP/GKE Note;

Make sure to add the bucket permissions to your service account. For example:

gsutil iam ch serviceAccount:${GCP_STORAGE_SA}:roles/storage.objectAdmin gs://my-bucket

{
    "type": "batch_import",
    "name": "job_sample_data",
    "description": "sample data on gcp",
    "settings": {
        "max_active_partitions": 576,
        "max_rows_per_partition": 33554432,
        "max_minutes_per_partition": 20,
        "source": {
            "settings": {
                "bucket_name": "burninbucket",  
                "bucket_path": "/gcp-prod-test",  
                "region": "us-east1",  
                "cloud": "gcp",  
                "credential_id": "11111111-1111-1111-1111-111111111111"
            },
            "table": "sample.data",
            "type": "batch",
            "transform": "mytransform"
        },
        "regex_filter": "^gs://burninbucket/gcp-prod-test/.*.gz"
    }
}

Linode Example

🚧

Limitations

The k8s cluster must use the same account as the Linode bucket. Linode storage does not support auto-ingest.

{
    "type": "batch_import",
    "name": "job_sample_data",
    "description": "sample data on Linode",
    "settings": {
        "max_active_partitions": 576,
        "max_rows_per_partition": 33554432,
        "max_minutes_per_partition": 20,
        "source": {
            "settings": {
                "bucket_name": "mydatatoingest",  
                "bucket_path": "/mypath",  
                "region": "us-southeast-1",  
                "cloud": "linode",  
                "credential_id": "11111111-1111-1111-1111-111111111111"
            },
            "table": "sample.data",
            "type": "batch",
            "transform": "mytransform"
        },
        "regex_filter": "^s3://mydatatoingest/mypath/.*.gz"
    }
}

Job Attributes

A job describes how to treat the data set as a whole as it is being ingested. Hydrolix batch jobs can employ varying file and path structures to load data. A single file, a directory of files, and a directory of files with a filter can all be applied.

For example:

• A single file i.e. "s3://mybucket/another/file.gz"
• All files in a single bucket i.e. "s3://mybucket/another/"
• All files matching regex pattern i.e. "s3://mybucket/" along with "settings.regex_filter": "^s3://mybucket/.*/.*.gz"

The settings object

Some data sets consist of many small files, other data sets consist of fewer larger files. Hydrolix ultimately writes data into "partitions". The number and size of partitions influences the performance of your queries.

What is best for each data set is an "it depends" answer, however, consider:

1. Partitions are a single unit to be processed. This means that queries of large partitions cannot be parallelized as much as smaller partitions.
2. Smaller partitions mean more parallelization, but also mean less efficient use of resources.

Example Settings:

{ ...
"settings": {
  	"max_active_partitions": 576,
		"max_rows_per_partition": 10000000,
		"max_minutes_per_partition": 14400,
    "input_concurrency": 1,
    "input_aggregation": 1536000000,
    "max_files": 0,
    "dry_run": false,
		"source": {
			...
		}
	}
}

The following are the default settings. We would suggest starting with the defaults and then tuning.

A note on Ingest Parallelization

Batch ingest is performed on compute instances. Batch performance can be improved by:

1. Increasing your batch-peer replica count
2. Vertically scaling your batch-peer instances
3. Increasing the number of max_active_partitions. This configuration sets a limit on the maximum number of partitions that a batch peer can process simultaneously.

Regex Filter

If data is stored in a complex bucket structure on cloud storage and cannot be expressed with a simple S3 path, regex_filter allows you to express the structure pattern to search.

Given the following example s3 source path
s3://mybucket/level1/2020/01/app1/pattern_xyz.gz

Possible regex_filter patterns could be:

• ^.*\\.gz$
• ^s3://mybucket/level1/2020/\\d{2}/app1/.*.gz
• ^.*/level1/2020/\\d{2}/app1/.*.gz
• ^.*/level1/2020/\\d{2}/.*/pattern_\\w{3}.gz
• ^.*/level1/2020/\\d{2}/.*/pattern_.*.gz

The source element

The source element specifies information about the data itself, where it is, and how it should be treated.

Example source:

{
    ...
        "source": {
          "table": "sample_project.sample_table",
          "type": "batch",
          "transform": "mytransform",
          "settings": {
            "bucket_name": "string",
            "bucket_path": "/",
            "region": "string",
            "endpoint": "string",
            "cloud": "string",
            "credential_id": "11111111-1111-1111-1111-111111111111"
					}
			}
    ...
}

Cancel Jobs

Use the cancel jobs endpoint to cancel the batch ingest job and tasks associated with the job ID. The cancellation will be reflected in the status output /v1/orgs/{org_id}/jobs/batch/{job_id}/cancel.

Jobs Status

Get the status of a job and its tasks. This endpoint is suitable for polling for job completion /v1/orgs/{org_id}/jobs/batch/{job_id}/status.

Job Response codes

Response body on success

{
  "status": "RUNNING",
  "status_detail": {
    "tasks": {
      "INDEX": {
        "READY": 5
      },
      "LIST": {
        "DONE": 1
      }
    },
    "percent_complete": 0.16666667,
    "estimated": false
  }
}