Skip to content

Bot Insights

Bot Insights is available in Hydrolix for AWS. It works with existing bot management tools that perform detection and blocking, answering three core questions:

  • What changed in bot traffic
  • What's driving the change
  • What it's costing

Bot Insights analyzes CDN log data across many vendors and surfaces results in five job-specific dashboards. It consolidates logs from any CDN vendor, including but not limited to:

  • CloudFront
  • Fastly
  • Cloudflare
  • Akamai
  • Tencent

See Bot Insights: Purpose-Built Dashboards for Every Team for a demonstration of this solution.

How it works⚓︎

Enabling Bot Insights generates the following infrastructure inside the Hydrolix cluster:

  • Ingest transforms for each CDN vendor
  • Dictionaries that the transforms reference
  • Summary tables
  • Grafana dashboards

During ingest into the Hydrolix cluster, each log record passes through a Bot Insights transform. The transform logic performs lookups in three dictionaries and derives the values for the various bot classification columns which are then surfaced in the Grafana dashboards. The cluster also maintains the hourly and daily summary tables that aggregate these dimensions to ensure low query latency at scale.

Prerequisites⚓︎

After adding Bot Insights, the detail page in the Hydrolix Web Console will show everything you need to complete configuration for each CDN.

Configure Bot Insights⚓︎

To add and configure the Bot Insights Solution:

  1. Sign in to the Hydrolix Web Console.
  2. In the left navigation, select Solutions & Data Sources.
  3. Select Add Solution or Data Source.
  4. Choose Bot Insights from the list.

After Hydrolix finishes provisioning, the Bot Insights detail page includes:

  • A list of supported CDN data sources with links to configuration instructions.
  • A link to the Bot Insights dashboards. Configure your CDN and optionally your WAF or bot manager to send logs to Hydrolix for these dashboards.
  • The name of any automatically provisioned resources.

Bot Insights appears on the Solutions & Data Sources page. Return to the Bot Insights detail page at any time to review configuration details or remove the solution.

Send CDN log data to Hydrolix⚓︎

You can send logs from any supported CDN at any time. Dashboards update as new log streams become available.

Adding a CDN data source in the Hydrolix Web Console completes all Hydrolix configuration. Follow the CDN instructions to send logs.

Configure AWS CloudFront logs⚓︎

Bot Insights uses CloudFront standard logs (v2).

When you add Bot Insights in the Hydrolix Web Console, Hydrolix automatically provisions an Amazon Data Firehose stream in your AWS account. To send logs:

  1. Follow the steps in the AWS documentation: Configure Standard Logging (v2) for CloudFront.

  2. When prompted to select a destination, choose the Hydrolix-created Firehose stream. The stream name appears on the Bot Insights detail page and follows the pattern:

    text Firehose stream name hdx-firehose-{random-string}

Configure Fastly streaming logs⚓︎

See Configure Fastly real-time log streaming to configure Fastly.

Cloudflare Logs⚓︎

See Configure Cloudflare Logpush to configure Cloudflare.

Configure Akamai DataStream logs⚓︎

See Create a Stream to configure Akamai DataStream.

Configure Tencent EdgeOne logs⚓︎

See Configure EdgeOne Real-Time Log Push to configure Tencent EdgeOne.

Dashboards⚓︎

Bot Insights includes five dashboards, each organized around specific tasks:

Dashboard Name Job
Overview and Alignment Is the data fresh, and is the situation real?
What Changed + Investigation What changed in automation, and what needs response?
SEO Governance Are good bots healthy, and are AI crawlers behaving?
Edge/Ops What are bots costing in infrastructure?
Multi-Domain Triage Who owns the issue, and does it need to be escalated?
SOC What Changed How bot pressure is impacting error rates, cache miss rates, and other key metrics?

Bot classification⚓︎

Bot Insights identifies 803 individual bots across 20 categories using Matomo Device Detector.

Using the device detector data, each bot is assigned two labels:

  • bot_class: A broad category such as a search crawler, AI crawler, or monitoring agent.
  • bot_intent: Hydrolix-derived value. A fine-grained intent within the bot class.

Dictionary sources⚓︎

External upstream sources provide the data in the three ClickHouse dictionaries used by Bot Insights to classify traffic.

  • Bot classifications (ua_cat_dict): The bot patterns and their categories come directly from Matomo Device Detector with some Hydrolix overlay as in the case of the bot_intent label.

Hydrolix handles updating the dictionaries from their respective sources. See Custom Dictionaries and Regexp Tree Dictionaries to read more about working with dictionaries in Hydrolix.

Evidence sources and capability tiers⚓︎

The value you get from Bot Insights depends on which data sources you connect. Three evidence sources feed the dashboards and combine into three capability tiers:

Supported CDNs Tier Data Value
Akamai (DS2 only), Tencent, Fastly, Cloudflare, CloudFront CDN Delivery + Dictionary Bot traffic mix, classification, volume trends, request paths
Akamai (DS2 + SIEM) CDN + Bot Manager Everything in CDN tier + Enforcement (Bot Manager) Vendor verdicts correlated with independent classification

Bot Insights and enforcement tools⚓︎

Bot Insights is an intelligence and visibility layer rather than an enforcement tool. It's designed to work alongside inline enforcement products such as bot managers:

Bot Manager Bot Insights
Enforcement: blocks or allows traffic in real time Analytics that reveal patterns over time
Optimized for low false positives. Lets some bot traffic through. Shows what got through and why
Reports what it blocked Determines what should be blocked but is making it through
Action-oriented Investigation-oriented

v1.0 vs v1.1⚓︎

Capability v1.0 v1.1
Bot identification 28 UA patterns (~80 bots) 803 bots via Matomo
Data sources CDN only CDN + WAF + Bot Manager
Dashboard structure Single overview Six job-specific dashboards with progressive disclosure

Bot Insights LLM skill⚓︎

The Hydrolix AI Toolkit includes a dedicated Bot Insights skill for use with many AI agents. See the toolkit installation instructions for supported agents. The skill enables agents to investigate bot behavior in Bot Insights data and generates evidence-backed reports tailored to different stakeholders.

Six report types serve different types of questions about bots

Report type Purpose
Executive posture High-level summary for executive and board alignment
Control review Validates whether bot management policy changes are producing intended results
SOC triage Security operations attribution and spoofing detection
Scorecard brief Entity scoring and reporting
Crawler governance Good-bot health and AI crawler compliance assessment
Edge/Ops impact Infrastructure cost and cache efficiency analysis