Tencent EdgeOne Integration

Overview

Tencent EdgeOne is a cloud-based edge platform that provides traffic acceleration and security features such as DDoS mitigation, a web application firewall (WAF), and bot management. It routes and processes user traffic at Tencent's global edge nodes to reduce latency, protect applications, and improve performance for content delivery and API traffic.

Setup steps

There are some prerequisites and three major steps covered within this document. Following these steps will result in an integrated setup between Tencent EdgeOne and a running Hydrolix cluster:

1. Before you Begin: Complete these prerequisites first
2. Configure a Hydrolix Transform: Prepare Hydrolix to receive and transform EdgeOne log data
3. Configure EdgeOne Real-Time Log Push: Set up Real-time Log Push in EdgeOne to send logs over HTTP to Hydrolix
4. Verification: Follow these steps to verify the integration is working correctly

Before you begin

These instructions use the EdgeOne Real-Time Log Push to send Tencent EdgeOne real-time logs directly to a Hydrolix cluster's HTTP Streaming API endpoint. The output columns specified in the Hydrolix transform correspond to EdgeOne log fields. The data flow is as follows:

Tencent Cloud Services → EdgeOne Real-Time Log Push → Hydrolix HTTP Streaming API

You'll need a Tencent Cloud account and a running Hydrolix deployment. Follow the instructions for your preferred deployment method if you have yet to deploy Hydrolix. From your running Hydrolix cluster, you will need the following information:

Keep these values on hand for later steps within this guide.

Getting started

You will be setting up two required components in these instructions to successfully integrate Tencent EdgeOne with a running Hydrolix cluster:

Configure a Hydrolix transform

You will need to register a Hydrolix transform. The transform defines a schema to map fields from the EdgeOne logs
onto the Hydrolix table.

Using the Hydrolix transform for Tencent EdgeOne, publish a transform for your EdgeOne log data using the instructions in Publishing Your Transform.

You have the option of registering a transform through the UI, which requires:

• Project name
• Table name
• The contents of the output_columns property from the transform

Alternatively, you can use the API, which requires:

• Project ID
• Table ID
• The entirety of the Tencent EdgeOne transform JSON

In either case, make sure you enable gzip compression as EdgeOne Log Push will gzip compress the logs before pushing them to Hydrolix.

Configure EdgeOne Real-Time Log Push

Follow the Push to HTTP Server instructions for EdgeOne Log Push to send logs to the Hydrolix cluster. You will need the following for the Destination Information Page:

They key/value pairs for the Custom HTTP header should at a minimum include the required headers for the HTTP Stream API endpoint of the Hydrolix cluster. For example:

Note that content-type isn't included, despite it being a required header, as it's a default header carried by EdgeOne log push. EdgeOne will also add the HTTP header Content-Encoding: gzip if you have compression enabled.

Alternatively, you can supply the Hydrolix table and transform as query parameters within the address. For example: https://{myhost}.hydrolix.live/ingest/event?table=project_name.table_name&transform=transform_name

Verification

To verify that the integration is working correctly:

1. Wait a few minutes for data to start flowing from EdgeOne to Hydrolix. Recall that the Logpush service is sending EdgeOne real-time logs to Hydrolix. If you don't see data in Hydrolix after a few minutes, confirm that EdgeOne is generating this type of data.
2. In the Hydrolix UI, navigate to the query interface. (https://{myhost}.hydrolix.live/queries/)
3. Run a simple query against your EdgeOne table. For example:

SELECT
  ClientState,
  EdgeSeverRegion,
  RequestID,
  RequestMethod,
  RequestUrl,
  EdgeResponseTime
FROM project_name.table_name
WHERE timestamp > NOW() - INTERVAL '1 hour'
LIMIT 100

4. Confirm that you see results similar to your data in EdgeOne.

Troubleshooting

If you encounter issues with the integration:

Verify your Hydrolix Bearer Boken is valid and hasn't expired

• If you receive a 401 Unauthorized response, then it's possible the bearer token expired. You can retrieve a new bearer token using the Hydrolix login API endpoint.

Review the ingest logs in Hydrolix for errors

• You can view and explore cluster logs using the cluster's System Health page.
• You can also look for errors in Hydrolix cluster logs using Hydrologs

Check ingest latency in table health

1. Navigate to https://{myhost}.hydrolix.live/data/tables/. Select your project and table. This will take you to a page which displays table health information.
2. Check that the value for Ingest Latency indicates a time since you last sent data from EdgeOne. Note that if the table is the destination for any other data streams, this number may be low due to data from another stream. If the latency indicates that no data has been received since the Epoch (January 1st, 1970 at 00:00:00 UTC), then the table has received no data.

Ensure your transform correctly maps the fields you expect to see

• The Hydrolix transform provided acts as a schema for storing EdgeOne log fields in Hydrolix. If you forward another data set to Hydrolix, see the Hydrolix documentation on transform output columns to modify the transform to correctly map the data you are sending.

Additional considerations

If you specified a custom log field for EdgeOne Log Push, you'll need to add an output field to the Hydrolix transform for it to be transformed and stored as a column. See Transform structure: output columns for more information.