Akamai WAF logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest WAF logs from Akamai. This article covers prerequisites, configuration instructions, connector schemas, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. An Akamai Control Center administrator must enable SIEM Integration and provision the SIEM user.
  4. The user associated with the API credentials must have the Manage SIEM role for every group containing a security configuration you want to ingest. See Akamai SIEM Integration.
  5. SIEM Integration and data collection must be turned on for each security configuration, and the configuration must be active on the production network. See Get started with the SIEM Integration API.

Known limitations

  • SCD type 2 is not supported, since Akamai WAF logs are append-only.
  • The connector authenticates using long-lived EdgeGrid credentials that default to 2 years. After they expire, create new credentials and update the connection.
  • Historical lookback can't exceed the 12 hours Akamai retains WAF logs. See Choose how far back to ingest.

Configure Akamai for ingestion

Configure Akamai to enable authentication from Databricks.

Fetch Web Security Configuration IDs

  1. Sign in to Akamai Control Center as an Admin.
  2. Click the menu icon in the top left of the page to open the navigation pane.
  3. In the navigation pane, click Your Services > Web & Data Center Security > Security Configurations > Web Security.
  4. Review the list of security configurations. Each configuration has a dropdown. Note the Contract and Group ID under the dropdown. You need these values when you create a user with the MANAGE_SIEM role.
  5. For each security configuration that you want to ingest logs for:
    1. Click the version under Last Created, Staging, or Production.
    2. Go to Advanced Settings > Logging > Data Collection For SIEM integration > Web Security Configuration ID.
    3. Note the Web Security Configuration ID. You need these values when you create a user with the MANAGE_SIEM role.

Create a user with the MANAGE_SIEM role

  1. Sign in to Akamai Control Center as an Admin.
  2. Click the menu icon in the top left of the page to open the navigation pane.
  3. In the navigation pane, click Your Services > Account Admin > Identity & Access.
  4. Click Create User.
  5. Enter the Email and Verify email values.
  6. Under Assign roles, select MANAGE_SIEM for each of the groups used to define the Web Security Configuration IDs from the previous step.

Create an API client

  1. Sign in to Akamai Control Center as the user you created in the previous step.
  2. Click the menu icon in the top left of the page to open the navigation pane.
  3. In the navigation pane, click Your Services > Account Admin > Identity & Access.
  4. Click Create API Client > Advanced.
  5. Under APIs, select Select APIs.
  6. Click Reset API selection.
  7. Search for the SIEM API.
  8. Under Access level, select READ-ONLY, then click Submit.
  9. Enter a unique Name for the API client, then click Create API Client.
  10. Copy the client_secret, host, access_token, and client_token values. Akamai shows these credentials only one time. Use them to create a Unity Catalog connection in Databricks.

Tip

When creating a connection through the Lakewatch UI, you need the Host, Client Token, Client Secret, and Access Token gathered above.

Next steps

After completing the configuration above, create a datasource with the Akamai WAF connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

The connector ingests the following source table from the default source schema:

Source table Akamai data Built-in Lakewatch parser
akamai_waf_events WAF security events from the Akamai SIEM Integration API akamai_waf

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds lw_id, a non-null string that Lakewatch uses for stable deduplication and time ordering. It derives lw_id from the event timestamp and _databricks_primary_key. The connector derives:

  • _databricks_primary_key from a stable hash of the source event.
  • time from httpMessage.start, interpreted as Unix epoch seconds. The connector uses time as the cursor field for incremental ingestion.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The following schema combines the standard Lakeflow Connect fields with the Lakewatch-only lw_id column. For field definitions, see the Akamai SIEM Integration API.

Schema
Column Data type
lw_id STRING
_databricks_primary_key LONG
time TIMESTAMP
format STRING
type STRING
version STRING
attackData STRUCT<appliedAction: STRING, clientIP: STRING, configId: STRING, policyId: STRING, ruleActions: STRING, ruleData: STRING, ruleMessages: STRING, ruleSelectors: STRING, ruleTags: STRING, ruleVersions: STRING, rules: STRING, apiId: STRING, apiKey: STRING, clientReputation: STRING, slowPostAction: STRING, slowPostRate: STRING, decodedRules: ARRAY<STRUCT<rule: STRING, ruleAction: STRING, ruleData: STRING, ruleMessage: STRING, ruleSelector: STRING, ruleTag: STRING, ruleVersion: STRING>>>
botData STRUCT<botScore: STRING, responseSegment: STRING>
clientData STRUCT<telemetryType: STRING, appBundleId: STRING, appVersion: STRING, sdkVersion: STRING>
custom STRING
geo STRUCT<asn: STRING, city: STRING, continent: STRING, country: STRING, regionCode: STRING>
httpMessage STRUCT<bytes: STRING, host: STRING, method: STRING, path: STRING, port: STRING, protocol: STRING, query: STRING, requestHeaders: STRING, requestId: STRING, responseHeaders: STRING, start: STRING, status: STRING, tls: STRING>
identity STRUCT<ja4: STRING, tlsFingerprintV2: STRING, tlsFingerprintV3: STRING>
userRiskData STRUCT<allow: STRING, emailDomain: STRING, general: STRING, originUserId: STRING, risk: STRING, score: STRING, status: STRING, trust: STRING, username: STRING, uuid: STRING>

Parser observables

The akamai_waf parser writes nested objects as VARIANT and extracts the following fields into lw_observables.

Observables
Field Observable
attackData.clientIP ip
httpMessage.host hostname
httpMessage.requestId trace_id
userRiskData.emailDomain domain
userRiskData.originUserId actor_id
userRiskData.username username
userRiskData.uuid actor_id
  • Last updated on