BytePlus

BytePlus is a real-time content delivery network (CDN) operated by ByteDance.

BytePlus supports streaming log delivery to a Hydrolix HTTP endpoint for points of presence (PoPs) outside of mainland China. For logs from PoPs within mainland China, you will need to work with your BytePlus representative to configure custom log delivery.

These logs contain structured fields such as request timestamps, client IPs, cache status, and response metrics, enabling visibility into CDN performance, reliability, and content delivery.

Follow the instructions on this page to stream BytePlus logs into Hydrolix.

Setup Steps

There are some prerequisites and three major steps covered in this document. Following these steps will result in an integrated setup between BytePlus and a running Hydrolix cluster:

1. Before you Begin: Complete these prerequisites first
2. Configure a Hydrolix Transform: Prepare Hydrolix to receive and transform BytePlus log data
3. Configure BytePlus log streaming: Add Hydrolix as a log target in BytePlus to send HTTP logs to Hydrolix
4. Verification: Follow these steps to verify the integration is working correctly

Before You Begin

You will need a BytePlus account and a running Hydrolix deployment. Follow the instructions for your preferred deployment method if you have yet to deploy Hydrolix. From your running Hydrolix cluster, you will need the following information:

Gather the following information:

Item	Description	Example value	How to obtain this information
Cluster ingest URL	This is the hostname and ingest endpoint of your Hydrolix cluster.	https://hostname.hydrolix.live/ingest/event	The value of hydrolix_url in your hydrolixcluster.yaml including the preceding https://.
Table name	The destination table in the Hydrolix cluster for BytePlus CDN logs. Specified in the format: project_name.table_name.	byteplus_project.byteplus_cdn_logs	See create a table if you need to create a Hydrolix table in which to store your BytePlus CDN logs.
Basic Authentication Credentials (base64 encoded)	The base64 encoded version of the username and password to authenticate with the Hydrolix endpoint and grant streaming ingest access to the destination table.	aW5nZXN0X3VzZXI6c2VjcmV0MTIz	For a more thorough explanation of basic authentication as it pertains to a Hydrolix cluster, see Enable Basic Authentication.

Keep these values on hand for later steps within this guide.

Get started

There are two steps covered within this document. Following these steps will result in an integrated setup between Fastly and a running Hydrolix cluster:

1. Create a Hydrolix transform
2. Configure BytePlus log streaming

Configure a Hydrolix transform

Register a Hydrolix transform. The transform defines a schema to map fields from the BytePlus logs onto the Hydrolix table.

Using the Hydrolix transform for BytePlus, see Publish Transform for instructions.

You have two options for the publishing the transform:

1. UI: Registering a transform through the UI requires the following information:

   • Project name
   • Table name
   • The contents of the output_columns property

2. API: Registering a transform through the API requires the following information:

   • Project ID
   • Table ID
   • The entirety of the BytePlus transform json

If the transform you have created is the only one for your table, it will become the default transform for the table. If you have multiple transforms configured for the destination table, and the transform registered for BytePlus isn't the default transform, you should specify the transform for this data as a query parameter in the endpoint URL.

Verify that the Hydrolix transform exists. You can do so using either method:

• UI: In the UI by navigating to Data in the left-hand nav, selecting the table from under the Tables tab, and locating the transform name under Table transforms.
• API: Verify the transform you created is returned as part of the response from the Get transforms endpoint.

Configure BytePlus log streaming

Follow these steps to configure BytePlus log streaming to send data to your Hydrolix cluster. You can also find instructions in the BytePlus documentation at Task Configuration: HTTP Endpoint:

1. From the BytePlus portal, select Create task
2. Select HTTP Endpoint as the Destination.

3. Fill out the form with these values:

4. Collection Region: Outside Chinese Mainland (for within Chinese Mainland, contact your BytePlus representative)

5. HTTP Endpoint: Enter the following URL, replacing the {} fields:

   https://{hostname}.hydrolix.live/ingest/event
   

6. Compression: Gzip

7. Require Authentication: Off
8. Sampling Rate: 100%

9. Set Request Headers: Add the following headers:

   • x-hdx-table: {project_name}.{table_name}
   • x-hdx-transform: {transform_name}
   • Authorization: Basic {Base64 encoding of username:password}

10. Select the Test button to test log delivery.

Verification

To verify that the integration is working correctly:

1. Wait a few minutes for data to start flowing from BytePlus to Hydrolix.
2. In the Hydrolix UI, navigate to the query interface. (https://hostname.hydrolix.live/queries/).

3. Run a simple query against your able. For example:

   SELECT
     timestamp,
     request_host,
     request_method,
     request_full_path,
     response_status_code,
     response_time_to_first_byte_ms
   FROM integrations.byteplus
   WHERE timestamp > NOW() - INTERVAL '1 hour'
   LIMIT 10
   

4. Verify that you see results similar to your data in BytePlus.

Going further

• For more information on querying your Hydrolix cluster, see ClickHouse SQL.
• Explore your BytePlus logs in Grafana dashboards.

Troubleshoot

If you encounter issues with the integration:

Review the ingest logs in Hydrolix for any errors

• You can view and explore cluster logs using the cluster's System Health page.

Check ingest latency in table health

• Navigate to https://hostname.hydrolix.live/data/tables/. Select your project and table. This will take you to a page which displays some table health information.
• Check that the value for Ingest Latency indicates a time since you last sent data from BytePlus. Note that if the table is the destination for any other data streams, this number may be low due to data from another stream. If the latency indicates a time since the epoch (January 1st, 1970 at 00:00:00 UTC), then the table has received no data.

Ensure your transform correctly maps the fields you expect to see

• The Hydrolix transform provided acts a schema for storing HTTP requests from BytePlus log fields in Hydrolix. If you forward another data set to Hydrolix, see the Hydrolix documentation on transform output columns to modify the transform to correctly map the data you are sending.