Akamai DataStream Integration
This guide explains how to ingest logs from Akamai's Datastream into Hydrolix using Stream ingestion.
Create a Stream
Let's begin by creating a stream in the Akamai portal.
-
In the Akamai admin portal, navigate to "COMMON SERVICES", then to "DataStream".
-
On the DataStream page, click "Create stream". You should see the following wizard:
-
In "Destination", select "Custom HTTPS".
-
Under "type", select "DataStream 2".
-
For the "Name" field, enter "Hydrolix".
-
For the endpoint URL, use
https://<yourhostname>/ingest/event?table=<yourproject.yourtable>&transform=akamai_transform. -
For "Authentication", select "basic".
-
Under "Custom Header", select
Content-Type:application/json. -
Check "send compressed data" to enable data compression in requests.
-
On the "Data sets" page, use the default options for all fields except for log format. For the log format, select "Json".
-
On the "Summary" page, click "Save stream" to create the stream.
Configure the Transform
By default, DataStream 2 sends the following fields in each message:
{
"version": 1,
"customField": "any-custom-value",
"reqId": "1239f220",
"reqTimeSec": "1573840000",
"city": "HERNDON",
"state": "Virginia",
"country": "US",
"cacheStatus": "1",
"breadcrumbs": "//BC/%5Ba=6.7.8.9,c=g,k=0,l=1%5D",
"dnsLookupTimeMSec": "50",
"transferTimeMSec": "125",
"turnAroundTimeMSec": "11",
"errorCode": "ERR_ACCESS_DENIED|fwd_acl",
"reqEndTimeMSec": "3",
"xForwardedFor": "1.2.3.4",
"maxAgeSec": "3600",
"referer": "https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest",
"range": "37334-42356",
"cookie": "cookie-content",
"accLang": "en-US",
"reqPort": "443",
"reqPath": "/path1/path2/file.ext",
"reqMethod": "GET",
"reqHost": "test.hostname.net",
"proto": "HTTPS",
"statusCode": "206",
"cliIP": "4.5.6.7",
"cp": "123456",
"bytes": "4995",
"rspContentLen": "5000",
"rspContentType": "text/html",
"UA": "Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29",
"tlsOverheadTimeMSec": "0",
"tlsVersion": "TLSv1",
"objSize": "484",
"uncompressedSize": "484",
"overheadBytes": "232",
"totalBytes": "0",
"queryStr": "param=value"
}
Separate Breadcrumbs into Discrete Columns
Breadcrumbs data arrives URL-encoded. We can use Hydrolix's built-in enrichment to decode breadcrumbs and separate out the different pieces of information contained within.
Consider the following example of breadcrumb data:
//BC/[a=6.7.8.9,c=g,k=0,l=1]
This string contains the following pieces of discrete information:
a=6.7.8.9is a component IPc=gis the edge ghostk=0is the request end time in millisecondsl=1is the turnaround time in milliseconds
Because you might want to query or monitor any of these fields individually, it makes sense to separate out each piece of data into a column in our table. Fortunately, Hydrolix enables you to enrich data with SQL statements at indexing time. Consider the following SQL statement that extracts the Edge IP:
SELECT extract(
extract(decodeURLComponent(breadcrumbs), '(\[[^[]*c=g[^]]*\])'),
'a=([^,\]]+)') AS Edge_IP
- The
decodeURLComponent()method translates the breadcrumbs from URL encoding into a string. - The
extract()method uses a regex to extract all of the data between the open and close brackets ([and]). - A second
extract()method call extracts the value aftera=. - We output the results to the
Edge_IPcolumn.
Enrichment SQL
You can use the following collection of SQL statements to decode all of the other fields:
SELECT now() - reqTimeSec as latency,
decodeURLComponent(assumeNotNull(UA)) as UA,
decodeURLComponent(assumeNotNull(state)) as state,
extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '\/\/BC\/(\S*)') as breadcrumbs,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'a=([^,\]]+)') as Edge_IP,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'b=([^,\]]+)') as Edge_RequestID,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'k=([^,\]]+)') as Edge_RequestEndTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'l=([^,\]]+)') as Edge_TurnAroundTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'm=([^,\]]+)') as Edge_DNSLookupTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'n=([^,\]]+)') as Edge_GeoInfo,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=g[^]]*\])'), 'o=([^,\]]+)') as Edge_ASN,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'a=([^,\]]+)') as Origin_IP,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'b=([^,\]]+)') as Origin_RequestID,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'k=([^,\]]+)') as Origin_RequestEndTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'l=([^,\]]+)') as Origin_TurnAroundTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'm=([^,\]]+)') as Origin_DNSLookupTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'n=([^,\]]+)') as Origin_GeoInfo,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\[[^[]*c=o[^]]*\])'), 'o=([^,\]]+)') as Origin_ASN,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'a=([^,\]]+)') as Peer_IP,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'b=([^,\]]+)') as Peer_RequestID,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'k=([^,\]]+)') as Peer_RequestEndTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'l=([^,\]]+)') as Peer_TurnAroundTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'm=([^,\]]+)') as Peer_DNSLookupTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'n=([^,\]]+)') as Peer_GeoInfo,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=p[^]]*\])'), 'o=([^,\]]+)') as Peer_ASN,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'a=([^,\]]+)') as Parent_IP,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'b=([^,\]]+)') as Parent_RequestID,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'k=([^,\]]+)') as Parent_RequestEndTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'l=([^,\]]+)') as Parent_TurnAroundTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'm=([^,\]]+)') as Parent_DNSLookupTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'n=([^,\]]+)') as Parent_GeoInfo,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=c[^]]*\])'), 'o=([^,\]]+)') as Parent_ASN,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'a=([^,\]]+)') as CloudWrapper_IP,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'b=([^,\]]+)') as CloudWrapper_RequestID,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'k=([^,\]]+)') as CloudWrapper_RequestEndTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'l=([^,\]]+)') as CloudWrapper_TurnAroundTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'm=([^,\]]+)') as CloudWrapper_DNSLookupTime,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'n=([^,\]]+)') as CloudWrapper_GeoInfo,
extract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\[[^[]*c=w[^]]*\])'), 'o=([^,\]]+)') as CloudWrapper_ASN,
* FROM {STREAM}
This results in the following set of columns and values for the above example:
{
"CloudWrapper_ASN": null,
"CloudWrapper_DNSLookupTime": null,
"CloudWrapper_GeoInfo": "",
"CloudWrapper_IP": "",
"CloudWrapper_RequestEndTime": null,
"CloudWrapper_RequestID": "",
"CloudWrapper_TurnAroundTime": null,
"Edge_ASN": null,
"Edge_DNSLookupTime": "",
"Edge_GeoInfo": "",
"Edge_IP": "6.7.8.9",
"Edge_RequestEndTime": 0,
"Edge_RequestID": "",
"Edge_TurnAroundTime": 1,
"Origin_ASN": null,
"Origin_DNSLookupTime": "",
"Origin_GeoInfo": "",
"Origin_IP": "",
"Origin_RequestEndTime": "",
"Origin_RequestID": "",
"Origin_TurnAroundTime": "",
"Parent_ASN": null,
"Parent_DNSLookupTime": null,
"Parent_GeoInfo": "",
"Parent_IP": "",
"Parent_RequestEndTime": null,
"Parent_RequestID": "",
"Parent_TurnAroundTime": null,
"Peer_ASN": null,
"Peer_DNSLookupTime": null,
"Peer_GeoInfo": "",
"Peer_IP": "",
"Peer_RequestEndTime": null,
"Peer_RequestID": "",
"Peer_TurnAroundTime": null,
"latency": 96,
"version": 1,
"customField": "any-custom-value",
"reqId": "1239f220",
"reqTimeSec": "2019-11-15 17:46:40",
"city": "HERNDON",
"state": "Virginia",
"country": "US",
"cacheStatus": "1",
"breadcrumbs": "[a=6.7.8.9,c=g,k=0,l=1]",
"dnsLookupTimeMSec": "50",
"transferTimeMSec": "125",
"turnAroundTimeMSec": "11",
"errorCode": "ERR_ACCESS_DENIED|fwd_acl",
"reqEndTimeMSec": "3",
"xForwardedFor": "1.2.3.4",
"maxAgeSec": "3600",
"referer": "https://test.referrer.net/en-US/docs/Web/test",
"range": "37334-42356",
"cookie": "cookie-content",
"accLang": "en-US",
"reqPort": "443",
"reqPath": "/path1/path2/file.ext",
"reqMethod": "GET",
"reqHost": "test.hostname.net",
"proto": "HTTPS",
"statusCode": "206",
"cliIP": "4.5.6.7",
"cp": "123456",
"bytes": "4995",
"rspContentLen": "5000",
"rspContentType": "text/html",
"UA": "Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_3)",
"tlsOverheadTimeMSec": "0",
"tlsVersion": "TLSv1",
"objSize": "484",
"uncompressedSize": "484",
"overheadBytes": "232",
"totalBytes": "0",
"queryStr": "param=value"
}
Example Transform
Before you can process data from Akamai, you'll need a transform to process the incoming data into your schema. The following transform handles DataStream 2 logs:
{
"name": "akamai_transform",
"description": "DataStream 2 Transform",
"settings": {
"is_default": true,
"sql_transform": "SELECT now() - reqTimeSec as latency,\ndecodeURLComponent(assumeNotNull(UA)) as UA,\ndecodeURLComponent(assumeNotNull(referer)) as referer,\ndecodeURLComponent(assumeNotNull(queryStr)) as queryStr,\nextract(decodeURLComponent(assumeNotNull(breadcrumbs)), '\\\\\\/\\\\\\/BC\\\\\\/(\\\\S*)') as breadcrumbs,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'a=([^,\\\\]]+)') as Edge_IP,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'b=([^,\\\\]]+)') as Edge_RequestID,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'k=([^,\\\\]]+)') as Edge_RequestEndTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'l=([^,\\\\]]+)') as Edge_TurnAroundTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'm=([^,\\\\]]+)') as Edge_DNSLookupTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'n=([^,\\\\]]+)') as Edge_GeoInfo,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=g[^]]*\\\\])'), 'o=([^,\\\\]]+)') as Edge_ASN,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'a=([^,\\\\]]+)') as Origin_IP,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'b=([^,\\\\]]+)') as Origin_RequestID,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'k=([^,\\\\]]+)') as Origin_RequestEndTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'l=([^,\\\\]]+)') as Origin_TurnAroundTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'm=([^,\\\\]]+)') as Origin_DNSLookupTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'n=([^,\\\\]]+)') as Origin_GeoInfo,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(\\\\[[^[]*c=o[^]]*\\\\])'), 'o=([^,\\\\]]+)') as Origin_ASN,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'a=([^,\\\\]]+)') as Peer_IP,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'b=([^,\\\\]]+)') as Peer_RequestID,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'k=([^,\\\\]]+)') as Peer_RequestEndTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'l=([^,\\\\]]+)') as Peer_TurnAroundTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'm=([^,\\\\]]+)') as Peer_DNSLookupTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'n=([^,\\\\]]+)') as Peer_GeoInfo,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=p[^]]*\\\\])'), 'o=([^,\\\\]]+)') as Peer_ASN,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'a=([^,\\\\]]+)') as Parent_IP,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'b=([^,\\\\]]+)') as Parent_RequestID,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'k=([^,\\\\]]+)') as Parent_RequestEndTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'l=([^,\\\\]]+)') as Parent_TurnAroundTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'm=([^,\\\\]]+)') as Parent_DNSLookupTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'n=([^,\\\\]]+)') as Parent_GeoInfo,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=c[^]]*\\\\])'), 'o=([^,\\\\]]+)') as Parent_ASN,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'a=([^,\\\\]]+)') as CloudWrapper_IP,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'b=([^,\\\\]]+)') as CloudWrapper_RequestID,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'k=([^,\\\\]]+)') as CloudWrapper_RequestEndTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'l=([^,\\\\]]+)') as CloudWrapper_TurnAroundTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'm=([^,\\\\]]+)') as CloudWrapper_DNSLookupTime,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'n=([^,\\\\]]+)') as CloudWrapper_GeoInfo,\nextract(extract(decodeURLComponent(assumeNotNull(breadcrumbs)), '(,\\\\[[^[]*c=w[^]]*\\\\])'), 'o=([^,\\\\]]+)') as CloudWrapper_ASN,\nassumeNotNull(decodeURLComponent(queryStr)) as url_cmcd,\nif(startsWith(url_cmcd, 'CMCD='), substring(url_cmcd, 6), url_cmcd) as url,\nflatten(extractAllGroupsVertical(assumeNotNull(url), '([^=,\\\"]+)=\\\"?([^,=\\\"]+)\\\"? ?')) as groups,\nakamai_cmcd_query_param('bl', groups) as cmcd_buffer_length,\nakamai_cmcd_query_param('br', groups) as cmcd_encoded_bitrate,\nakamai_cmcd_query_param('bs', groups) as cmcd_buffer_starvation,\nakamai_cmcd_query_param('cid', groups) as cmcd_content_id,\nakamai_cmcd_query_param('d', groups) as cmcd_object_duration,\nakamai_cmcd_query_param('dl', groups) as cmcd_deadline,\nakamai_cmcd_query_param('mtp', groups) as cmcd_measured_throughput,\nakamai_cmcd_query_param('nor', groups) as cmcd_next_object_requests,\nakamai_cmcd_query_param('nrr', groups) as cmcd_next_range_request,\nakamai_cmcd_object_type(akamai_cmcd_query_param('ot', groups)) as cmcd_object_type,\nakamai_cmcd_query_param('pr', groups) as cmcd_playback_rate,\nakamai_cmcd_query_param('rtp', groups) as cmcd_requested_max_throughput,\nakamai_cmcd_streaming_format(akamai_cmcd_query_param('sf', groups)) as cmcd_streaming_format,\nakamai_cmcd_query_param('sid', groups) as cmcd_session_id,\nakamai_cmcd_stream_type(akamai_cmcd_query_param('st', groups)) as cmcd_stream_type,\nakamai_cmcd_query_param('su', groups) as cmcd_startup,\nakamai_cmcd_query_param('tb', groups) as cmcd_top_bitrate,\nakamai_cmcd_query_param('v', groups) as cmcd_version,\nflatten(extractAllGroups(assumeNotNull(queryStr),'([^&]+)')) as extracted,\narrayMap(v -> splitByChar('=', v), extracted) as splitted,\narrayMap(y -> (y[1]), splitted) as key,\narrayMap(y -> (y[2]), splitted) as value,\n(key, value)::Map(String, String) AS querystring,\n* FROM {STREAM}",
"null_values": [
"-",
"^",
"-1"
],
"output_columns": [
{
"name": "version",
"datatype": {
"type": "uint8"
}
},
{
"name": "streamId",
"datatype": {
"type": "uint32"
}
},
{
"name": "customField",
"datatype": {
"type": "string"
}
},
{
"name": "reqId",
"datatype": {
"type": "string"
}
},
{
"name": "reqTimeSec",
"datatype": {
"type": "epoch",
"index": false,
"primary": true,
"source": null,
"format": "s",
"resolution": "s",
"default": null,
"script": null
}
},
{
"name": "city",
"datatype": {
"type": "string"
}
},
{
"name": "state",
"datatype": {
"type": "string"
}
},
{
"name": "country",
"datatype": {
"type": "string"
}
},
{
"name": "serverCountry",
"datatype": {
"type": "string"
}
},
{
"name": "billingRegion",
"datatype": {
"type": "uint32"
}
},
{
"name": "cacheStatus",
"datatype": {
"type": "boolean"
}
},
{
"name": "breadcrumbs",
"datatype": {
"type": "string"
}
},
{
"name": "securityRules",
"datatype": {
"type": "string"
}
},
{
"name": "ewUsageInfo",
"datatype": {
"type": "string"
}
},
{
"name": "ewExecutionInfo",
"datatype": {
"type": "string"
}
},
{
"name": "dnsLookupTimeMSec",
"datatype": {
"type": "uint32"
}
},
{
"name": "lastByte",
"datatype": {
"type": "boolean"
}
},
{
"name": "edgeIP",
"datatype": {
"type": "string"
}
},
{
"name": "transferTimeMSec",
"datatype": {
"type": "uint32"
}
},
{
"name": "turnAroundTimeMSec",
"datatype": {
"type": "uint32"
}
},
{
"name": "errorCode",
"datatype": {
"type": "string"
}
},
{
"name": "reqEndTimeMSec",
"datatype": {
"type": "uint32"
}
},
{
"name": "xForwardedFor",
"datatype": {
"type": "string"
}
},
{
"name": "maxAgeSec",
"datatype": {
"type": "uint64"
}
},
{
"name": "referer",
"datatype": {
"type": "string"
}
},
{
"name": "range",
"datatype": {
"type": "string"
}
},
{
"name": "cookie",
"datatype": {
"type": "string",
"index": true,
"index_options": {
"fulltext": true
}
}
},
{
"name": "accLang",
"datatype": {
"type": "string"
}
},
{
"name": "reqPort",
"datatype": {
"type": "uint32"
}
},
{
"name": "reqPath",
"datatype": {
"type": "string",
"index": true,
"index_options": {
"fulltext": true
}
}
},
{
"name": "reqMethod",
"datatype": {
"type": "string"
}
},
{
"name": "reqHost",
"datatype": {
"type": "string"
}
},
{
"name": "proto",
"datatype": {
"type": "string"
}
},
{
"name": "statusCode",
"datatype": {
"type": "uint32"
}
},
{
"name": "cliIP",
"datatype": {
"type": "string"
}
},
{
"name": "cp",
"datatype": {
"type": "uint32"
}
},
{
"name": "bytes",
"datatype": {
"type": "uint64"
}
},
{
"name": "rspContentLen",
"datatype": {
"type": "uint64"
}
},
{
"name": "rspContentType",
"datatype": {
"type": "string"
}
},
{
"name": "UA",
"datatype": {
"type": "string",
"index": true,
"index_options": {
"fulltext": true
}
}
},
{
"name": "tlsOverheadTimeMSec",
"datatype": {
"type": "uint32"
}
},
{
"name": "tlsVersion",
"datatype": {
"type": "string"
}
},
{
"name": "objSize",
"datatype": {
"type": "uint64"
}
},
{
"name": "uncompressedSize",
"datatype": {
"type": "uint64"
}
},
{
"name": "overheadBytes",
"datatype": {
"type": "uint64"
}
},
{
"name": "totalBytes",
"datatype": {
"type": "uint64"
}
},
{
"name": "queryStr",
"datatype": {
"type": "string"
}
},
{
"name": "Edge_IP",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_RequestID",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_RequestEndTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_TurnAroundTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_DNSLookupTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_GeoInfo",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Edge_ASN",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_IP",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_RequestID",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_RequestEndTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_TurnAroundTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_DNSLookupTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_GeoInfo",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Origin_ASN",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_IP",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_RequestID",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_RequestEndTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_TurnAroundTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_DNSLookupTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_GeoInfo",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Parent_ASN",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_IP",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_RequestID",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_RequestEndTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_TurnAroundTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_DNSLookupTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_GeoInfo",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "CloudWrapper_ASN",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_IP",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_RequestID",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_RequestEndTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_TurnAroundTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_DNSLookupTime",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_GeoInfo",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "Peer_ASN",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "latency",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_buffer_length",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_encoded_bitrate",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_buffer_starvation",
"datatype": {
"type": "uint8",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_content_id",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_object_duration",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_deadline",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_measured_throughput",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_next_object_requests",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_next_range_request",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_object_type",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_playback_rate",
"datatype": {
"type": "uint8",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_requested_max_throughput",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_streaming_format",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_session_id",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_stream_type",
"datatype": {
"type": "string",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_startup",
"datatype": {
"type": "uint8",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_top_bitrate",
"datatype": {
"type": "uint64",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "cmcd_version",
"datatype": {
"type": "uint32",
"source": {
"from_input_field": "sql_transform"
}
}
},
{
"name": "querystring",
"datatype": {
"type": "map",
"index": true,
"source": {
"from_input_field": "sql_transform"
},
"elements": [
{
"type": "string",
"index": true
},
{
"type": "string",
"index": true
}
]
}
},
{
"name": "isMidgress",
"datatype": {
"type": "boolean"
}
},
{
"name": "unknown",
"datatype": {
"type": "map",
"index": true,
"catch_all": true,
"elements": [
{
"type": "string",
"index": true
},
{
"type": "string",
"index": true
}
]
}
}
],
"compression": "none"
},
"type": "json"
}
Updated 29 days ago