Skip to content

Bot Insights

Bot Insights is available in Hydrolix for AWS. It works with existing bot management and enforcement tools such as AWS WAF and PerimeterX / Human Bot Defender. These tools perform detection and blocking, while Bot Insights answers three core questions:

  • What changed in bot traffic
  • What's driving the change
  • What it's costing

Bot Insights analyzes CDN log data, WAF data, and bot manager enforcement logs across many vendors and surfaces results in five job-specific dashboards.

Bot Insights consolidates logs from any CDN and Bot Manager including but not limited to:

CDN WAF and Bot Management
CloudFront AWS WAF Bot Control
Fastly Fastly Next-Gen WAF
Cloudflare Cloudflare Bot Management
Akamai Akamai Bot Manager
PerimeterX / Human Bot Defender

See Bot Insights: Purpose-Built Dashboards for Every Team for a demonstration of this solution.

How it works⚓︎

Enabling Bot Insights generates the following infrastructure inside the Hydrolix cluster:

  • Ingest transforms for each CDN vendor
  • Dictionaries that the transforms reference
  • Summary tables
  • Grafana dashboards

During ingest into the Hydrolix cluster, each log record passes through a Bot Insights transform. The transform logic performs lookups in three dictionaries and derives the values for the various bot classification columns which are then surfaced in the Grafana dashboards. The cluster also maintains the hourly and daily summary tables that aggregate these dimensions to ensure low query latency at scale.

Prerequisites⚓︎

After adding Bot Insights, the detail page in the Hydrolix Web Console will show everything you need to complete configuration for each CDN.

Configure Bot Insights⚓︎

To add and configure the Bot Insights Solution:

  1. Sign in to the Hydrolix Web Console.
  2. In the left navigation, select Solutions & Data Sources.
  3. Select Add Solution or Data Source.
  4. Choose Bot Insights from the list.

After Hydrolix finishes provisioning, the Bot Insights detail page includes:

  • A list of supported CDN data sources with links to configuration instructions.
  • A link to the Bot Insights dashboards. Configure your CDN and optionally your WAF or bot manager to send logs to Hydrolix for these dashboards.
  • The name of any automatically provisioned resources.

Bot Insights appears on the Solutions & Data Sources page. Return to the Bot Insights detail page at any time to review configuration details or remove the solution.

Send CDN log data to Hydrolix⚓︎

You can send logs from any supported CDN at any time. Dashboards update as new log streams become available.

Adding a CDN data source in the Hydrolix Web Console completes all Hydrolix configuration. Follow the CDN instructions to send logs.

Configure AWS CloudFront logs⚓︎

Bot Insights uses CloudFront standard logs (v2).

When you add Bot Insights in the Hydrolix Web Console, Hydrolix automatically provisions an Amazon Data Firehose stream in your AWS account. To send logs:

  1. Follow the steps in the AWS documentation: Configure Standard Logging (v2) for CloudFront.

  2. When prompted to select a destination, choose the Hydrolix-created Firehose stream. The stream name appears on the Bot Insights detail page and follows the pattern:

    text Firehose stream name hdx-firehose-{random-string}

Configure Fastly streaming logs⚓︎

See Configure Fastly real-time log streaming to configure Fastly.

Cloudflare Logs⚓︎

See Configure Cloudflare Logpush to configure Cloudflare.

Configure Akamai DataStream logs⚓︎

See Create a Stream to configure Akamai DataStream.

Dashboards⚓︎

Bot Insights includes five dashboards, each organized around specific tasks:

Dashboard Name Job
Overview and Alignment Is the data fresh, and is the situation real?
What Changed + Investigation What changed in automation, and what needs response?
Crawler Impact + Governance Are good bots healthy, and are AI crawlers behaving?
Origin Cost + Cache Efficiency What are bots costing in infrastructure?
Where's the Problem? Who owns the issue, and does it need to be escalated?

Bot classification⚓︎

Bot Insights identifies 803 individual bots across 20 categories using Matomo Device Detector.

Using the device detector data, each bot is assigned two labels:

  • bot_class: A broad category such as a search crawler, AI crawler, or monitoring agent.
  • bot_intent: Hydrolix-derived value. A fine-grained intent within the bot class.

Dictionary sources⚓︎

External upstream sources provide the data in the three ClickHouse dictionaries used by Bot Insights to classify traffic.

  • Bot classifications (ua_cat_dict): The bot patterns and their categories come directly from Matomo Device Detector with some Hydrolix overlay as in the case of the bot_intent label.
  • IP prefixes (bot_ip_dict): Pulled from vendor-published endpoints listing each vendor's bot CIDR ranges, including Google, Bing, OpenAI, and Amazon.
  • ASN reputation and type data (asn_type_dict): Fetched from the ipapi.is ASN classification database which maps ~80,000 ASNs to a type such as hosting, isp, business, education, or government.

Hydrolix handles updating the dictionaries from their respective sources. See Custom Dictionaries and Regexp Tree Dictionaries to read more about working with dictionaries in Hydrolix.

Verification signals⚓︎

Bot Insights identifies bots using three signals:

  1. User agent patterns
  2. IP CIDR ranges published by bot operators
  3. ASN reputation

Verification results in two additional fields on each request: bot_verification_tier and bot_confidence. Bot Insights flags traffic that can't be matched or verified due to, for example, bots that misrepresent their identity.

Evidence sources and capability tiers⚓︎

The value you get from Bot Insights depends on which data sources you connect. Three evidence sources feed the dashboards and combine into three capability tiers:

Tier Data Value
CDN Delivery + Dictionary Bot traffic mix, classification, volume trends
CDN + WAF + Enforcement (WAF) Request paths, API abuse, robots.txt compliance
CDN + WAF + Bot Manager + Enforcement (Bot Manager) Vendor verdicts correlated with independent classification

Bot Insights and enforcement tools⚓︎

Bot Insights is an intelligence and visibility layer rather than an enforcement tool. It's designed to work alongside inline enforcement products such as bot managers:

Bot Manager Bot Insights
Enforcement: blocks or allows traffic in real time Analytics that reveal patterns over time
Optimized for low false positives. Lets some bot traffic through. Shows what got through and why
Reports what it blocked Determines what should be blocked but is making it through
Single-vendor classification Multi-signal independent classification
Action-oriented Investigation-oriented

v1.0 vs v1.1⚓︎

Capability v1.0 v1.1
Bot identification 28 UA patterns (~80 bots) 803 bots via Matomo + three-signal verification
Classification depth Single label Two labels: bot_class + bot_intent
Spoof detection None IP CIDR + ASN reputation verification
Data sources CDN only CDN + WAF + Bot Manager
Dashboard structure Single overview Five job-specific dashboards with progressive disclosure
WAF integration Adjacent panels Coalesced panels with graceful degradation

Bot Insights LLM skill⚓︎

The Hydrolix AI Toolkit includes a dedicated Bot Insights skill for use with many AI agents. See the toolkit installation instructions for supported agents. The skill enables agents to investigate bot behavior in Bot Insights data and generates evidence-backed reports tailored to different stakeholders.

Six report types serve different types of questions about bots

Report type Purpose
Executive posture High-level summary for executive and board alignment
Control review Validates whether bot management policy changes are producing intended results
SOC triage Security operations attribution and spoofing detection
Scorecard brief Entity scoring and reporting
Crawler governance Good-bot health and AI crawler compliance assessment
Edge/Ops impact Infrastructure cost and cache efficiency analysis