AWS CloudTrail

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Overview

AWS CloudTrail records account activity and API usage across your AWS infrastructure. Lakewatch ingests CloudTrail logs from cloud storage (Amazon S3) using Auto Loader and schematizes them with built-in parsers. CloudTrail is a cloud storage source, not an API connector.

Onboard CloudTrail logs

To ingest CloudTrail, create a cloud-storage datasource that reads your CloudTrail logs from Amazon S3, and attach the built-in parser that matches the CloudTrail objects you're ingesting so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

CloudTrail delivers three types of objects to Amazon S3, and Lakewatch provides a built-in parser for each. Attach the parser that matches the objects your datasource ingests:

• aws_cloudtrail: Standard CloudTrail management, data, and network activity events.
• aws_cloudtrail_digest: Digest files, which list the log files delivered in the previous hour with hash values for log-file integrity validation.
• aws_cloudtrail_insight: CloudTrail Insights events, which record unusual API call-rate activity.

Every parsed table also carries the Lakewatch-managed columns:

• lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
• time: A non-null event timestamp.
• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see the Parser reference.

aws_cloudtrail

Schematizes standard CloudTrail events. Parses time from the eventTime field.

Parsed table schema

The aws_cloudtrail parser writes the following columns (generated from the parser definition):

Column	Data type
additionalEventData	VARIANT
apiVersion	STRING
awsRegion	STRING
errorCode	STRING
errorMessage	STRING
eventID	STRING
eventName	STRING
eventSource	STRING
time	TIMESTAMP
eventType	STRING
eventVersion	STRING
managementEvent	BOOLEAN
readOnly	BOOLEAN
recipientAccountId	STRING
requestID	STRING
requestParameters	VARIANT
resources	ARRAY<STRUCT<arn: STRING, accountId: STRING, type: STRING>>
responseElements	VARIANT
serviceEventDetails	VARIANT
sharedEventID	STRING
sourceIPAddress	STRING
userAgent	STRING
userIdentity	STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, accessKeyId: STRING, userName: STRING, sessionContext: STRUCT<attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>, sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, userName: STRING>, webIdFederationData: STRUCT<federatedProvider: STRING, attributes: VARIANT>, ec2RoleDelivery: STRING, sourceIdentity: STRING>, invokedBy: STRING, identityProvider: STRING, onBehalfOf: STRUCT<userId: STRING, identityStoreArn: STRING>, inScopeOf: STRUCT<sourceArn: STRING, sourceAccount: STRING, issuerType: STRING, credentialsIssuedTo: STRING>, credentialId: STRING>
vpcEndpointId	STRING
eventCategory	STRING
sessionCredentialFromConsole	BOOLEAN
edgeDeviceDetails	VARIANT
tlsDetails	STRUCT<tlsVersion: STRING, cipherSuite: STRING, clientProvidedHostHeader: STRING>
addendum	STRUCT<reason: STRING, updatedFields: STRING, originalRequestID: STRING, originalEventID: STRING>

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
recipientAccountId	aws_account_id
resources[].arn	aws_arn
resources[].accountId	aws_account_id
sourceIPAddress	hostname
userIdentity.principalId	actor_id
userIdentity.arn	aws_arn
userIdentity.accountId	aws_account_id
userIdentity.accessKeyId	trace_id
userIdentity.userName	username
userIdentity.sessionContext.sessionIssuer.principalId	actor_id
userIdentity.sessionContext.sessionIssuer.arn	aws_arn
userIdentity.sessionContext.sessionIssuer.accountId	aws_account_id
userIdentity.sessionContext.sessionIssuer.userName	username
userIdentity.sessionContext.webIdFederationData.federatedProvider	aws_arn
userIdentity.onBehalfOf.identityStoreArn	aws_arn
userIdentity.inScopeOf.sourceArn	aws_arn
userIdentity.inScopeOf.sourceAccount	aws_account_id

The following VARIANT fields are scanned for observables by matching nested key names. The observable types extracted from each:

Field	Observable types scanned
additionalEventData	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
requestParameters	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
responseElements	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
serviceEventDetails	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
userIdentity.sessionContext.webIdFederationData.attributes	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username

aws_cloudtrail_digest

Schematizes CloudTrail digest files, which support log-file integrity validation. Parses time from the digestEndTime field.

Parsed table schema

The aws_cloudtrail_digest parser writes the following columns (generated from the parser definition):

Column	Data type
awsAccountId	STRING
digestStartTime	TIMESTAMP
digestS3Bucket	STRING
digestS3Object	STRING
newestEventTime	TIMESTAMP
oldestEventTime	TIMESTAMP
previousDigestS3Bucket	STRING
previousDigestS3Object	STRING
previousDigestHashValue	STRING
previousDigestHashAlgorithm	STRING
previousDigestSignature	STRING
digestPublicKeyFingerprint	STRING
digestSignatureAlgorithm	STRING
logFiles	ARRAY<STRUCT<s3Bucket: STRING, s3Object: STRING, hashValue: STRING, hashAlgorithm: STRING, newestEventTime: TIMESTAMP, oldestEventTime: TIMESTAMP>>
time	TIMESTAMP

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
awsAccountId	aws_account_id
previousDigestHashValue	hash
logFiles[].hashValue	hash

aws_cloudtrail_insight

Schematizes CloudTrail Insights events, which record unusual API call-rate activity. Parses time from the eventTime field.

Parsed table schema

The aws_cloudtrail_insight parser writes the following columns (generated from the parser definition):

Column	Data type
eventVersion	STRING
awsRegion	STRING
eventID	STRING
eventType	STRING
recipientAccountId	STRING
sharedEventID	STRING
insightDetails	STRUCT<state: STRING, eventSource: STRING, eventName: STRING, insightType: STRING, insightContext: STRUCT<statistics: STRUCT<baseline: STRUCT<average: DOUBLE>, insight: STRUCT<average: DOUBLE>, insightDuration: DOUBLE>>>
eventCategory	STRING
time	TIMESTAMP

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
recipientAccountId	aws_account_id
sharedEventID	trace_id