Skip to content

Bot Insights

Overview⚓︎

Bot Insights is available in Hydrolix for AWS. It provides visibility into how different types of bots interact with your web properties by analyzing your CDN log data. The results are displayed in a single dashboard to inform bot management, with a focus on AI bots.

Adding the solution provisions the required ingestion streams and dashboards. Once added, configure the CDN to send logs to the Hydrolix service.

Bot Insights consolidates logs from these CDNs:

  • CloudFront
  • Fastly
  • Cloudflare
  • Akamai DataStream

See Bot Insights: Strategic Bot Management for the AI Era for a demonstration of this solution.

Prerequisites⚓︎

After adding Bot Insights, the detail page in the Hydrolix Web Console will show everything you need to complete configuration for each CDN.

Configure Bot Insights⚓︎

To add and configure the Bot Insights Solution:

  1. Sign in to the Hydrolix Web Console.
  2. In the left navigation, select Solutions & Data Sources.
  3. Select Add Solution or Data Source.
  4. Choose Bot Insights from the list.

After Hydrolix finishes provisioning, the Bot Insights detail page includes:

  • A list of supported CDN data sources with links to configuration instructions.
  • A link to the Bot Insights dashboard.
  • The name of any automatically provisioned resources.

Bot Insights appears on the Solutions & Data Sources page. Return to the Bot Insights detail page at any time to review configuration details or remove the solution.

Send log data to Hydrolix⚓︎

You can send logs from any supported CDN at any time. Dashboards update as new log streams become available.

Adding a CDN data source in the Hydrolix Web Console completes all Hydrolix configuration. Follow the CDN instructions to send logs.

Configure AWS CloudFront logs⚓︎

Bot Insights uses CloudFront standard logs (v2).

When you add Bot Insights in the Hydrolix Web Console, Hydrolix automatically provisions an Amazon Data Firehose stream in your AWS account. To send logs:

  1. Follow the steps in the AWS documentation: Configure Standard Logging (v2) for CloudFront.

  2. When prompted to select a destination, choose the Hydrolix-created Firehose stream. The stream name appears on the Bot Insights detail page and follows the pattern:

    text Firehose stream name hdx-firehose-{random-string}

Configure Fastly streaming logs⚓︎

See Configure Fastly real-time log streaming to configure Fastly.

Cloudflare Logs⚓︎

See Configure Cloudflare Logpush to configure Cloudflare.

Configure Akamai DataStream logs⚓︎

See Create a Stream to configure Akamai DataStream.