Security Control: Logging and threat detection

  • Article

Logging and Threat Detection covers controls for detecting threats on cloud, and enabling, collecting, and storing audit logs for cloud services, including enabling detection, investigation, and remediation processes with controls to generate high-quality alerts with native threat detection in cloud services; it also includes collecting logs with a cloud monitoring service, centralizing security analysis with a SIEM, time synchronization, and log retention.

LT-1: Enable threat detection capabilities

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.11 AU-3, AU-6, AU-12, SI-4 10.6, 10.8, A3.5

Security principle: To support threat detection scenarios, monitor all known resource types for known and expected threats and anomalies. Configure your alert filtering and analytics rules to extract high-quality alerts from log data, agents, or other data sources to reduce false positives.

Azure guidance: Use the threat detection capability of Microsoft Defender for Cloud for the respective Azure services.

For threat detection not included in Microsoft Defender services, refer to Microsoft Cloud Security Benchmark service baselines for the respective services to enable the threat detection or security alert capabilities within the service. Ingest alerts and log data from Microsoft Defender for Cloud, Microsoft 365 Defender, and log data from other resources into your Azure Monitor or Microsoft Sentinel instances to build analytics rules, which detect threats and create alerts that match specific criteria across your environment.

For Operational Technology (OT) environments that include computers that control or monitor Industrial Control System (ICS) or Supervisory Control and Data Acquisition (SCADA) resources, use Microsoft Defender for IoT to inventory assets and detect threats and vulnerabilities.

For services that do not have a native threat detection capability, consider collecting the data plane logs and analyze the threats through Microsoft Sentinel.

Azure implementation and additional context:

AWS guidance: Use Amazon GuardDuty for threat detection which analyzes and processes the following data sources: VPC Flow Logs, AWS CloudTrail management event logs, CloudTrail S3 data event logs, EKS audit logs, and DNS logs. GuardDuty is capable of reporting on security issues such as privilege escalation, exposed credential usage , or communication with malicious IP addresses, or domains.

Configure AWS Config to check rules in SecurityHub for compliance monitoring such as configuration drift, and create findings when needed.

For threat detection not included in GuardDuty and SecurityHub, enable threat detection or security alert capabilities within the supported AWS services. Extract the alerts to your CloudTrail, CloudWatch, or Microsoft Sentinel to build analytics rules, which hunt threats that match specific criteria across your environment.

You can also use Microsoft Defender for Cloud to monitor certain services in AWS such as EC2 instances.

For Operational Technology (OT) environments that include computers that control or monitor Industrial Control System (ICS) or Supervisory Control and Data Acquisition (SCADA) resources, use Microsoft Defender for IoT to inventory assets and detect threats and vulnerabilities.

AWS implementation and additional context:

GCP guidance: Use the Event Threat Detection in Google Cloud Security Command Center for threat detection using log data such as Admin Activity, GKE Data Access, VPC Flow Logs, Cloud DNS, and Firewall Logs.

Additionally use the Security Operations suite for the modern SOC with Chronicle SIEM and SOAR. Chronicle SIEM and SOAR provides threat detection, investigation, and hunting capabilities

You can also use Microsoft Defender for Cloud to monitor certain services in GCP such as Compute VM instances.

For Operational Technology (OT) environments that include computers that control or monitor Industrial Control System (ICS) or Supervisory Control and Data Acquisition (SCADA) resources, use Microsoft Defender for IoT to inventory assets and detect threats and vulnerabilities.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-2: Enable threat detection for identity and access management

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.11 AU-3, AU-6, AU-12, SI-4 10.6, 10.8, A3.5

Security principle: Detect threats for identities and access management by monitoring the user and application sign-in and access anomalies. Behavioral patterns such as excessive number of failed login attempts, and deprecated accounts in the subscription, should be alerted.

Azure guidance: Azure AD provides the following logs that can be viewed in Azure AD reporting or integrated with Azure Monitor, Microsoft Sentinel or other SIEM/monitoring tools for more sophisticated monitoring and analytics use cases:

  • Sign-ins: The sign-ins report provides information about the usage of managed applications and user sign-in activities.
  • Audit logs: Provides traceability through logs for all changes done by various features within Azure AD. Examples of audit logs include changes made to any resources within Azure AD like adding or removing users, apps, groups, roles and policies.
  • Risky sign-ins: A risky sign-in is an indicator for a sign-in attempt that might have been performed by someone who is not the legitimate owner of a user account.
  • Users flagged for risk: A risky user is an indicator for a user account that might have been compromised.

Azure AD also provides an Identity Protection module to detect and remediate risks related to user accounts and sign-in behaviors. Examples of risks include leaked credentials, sign-in from anonymous or malware linked IP addresses, password spray. The policies in Azure AD Identity Protection allow you to enforce risk-based MFA authentication in conjunction with Azure Conditional Access on user accounts.

In addition, Microsoft Defender for Cloud can be configured to alert on deprecated accounts in the subscription and suspicious activities such as an excessive number of failed authentication attempts. In addition to the basic security hygiene monitoring, Microsoft Defender for Cloud's Threat Protection module can also collect more in-depth security alerts from individual Azure compute resources (such as virtual machines, containers, app service), data resources (such as SQL DB and storage), and Azure service layers. This capability allows you to see account anomalies inside the individual resources.

Note: If you are connecting your on-premises Active Directory for synchronization, use the Microsoft Defender for Identity solution to consume your on-premises Active Directory signals to identify, detect, and investigate advanced threats, compromised identities, and malicious insider actions directed at your organization.

Azure implementation and additional context:

AWS guidance: AWS IAM provides the following reporting the logs and reports for console user activities through IAM Access Advisor and IAM credential report:

  • Every successful sign-in and unsuccessful login attempts.
  • Multi-factor authentication (MFA) status for each user.
  • Dormant IAM user

For API level access monitoring and threat detection, use Amazon GuadDuty to identify the findings related to the IAM. Examples of these findings include:

  • An API used to gain access to an AWS environment and was invoked in an anomalous way, or was used to evade defensive measures
  • An API used to:
    • discover resources was invoked in an anomalous way
    • collect data from an AWS environment was invoked in an anomalous way.
    • tamper with data or processes in an AWS environment was invoked in an anomalous way.
    • gain unauthorized access to an AWS environment was invoked in an anomalous way.
    • maintain unauthorized access to an AWS environment was invoked in an anomalous way.
    • obtain high-level permissions to an AWS environment was invoked in an anomalous way.
    • be invoked from a known malicious IP address.
    • be invoked using root credentials.
  • AWS CloudTrail logging was disabled.
  • Account password policy was weakened.
  • Multiple worldwide successful console logins were observed.
  • Credentials that were created exclusively for an EC2 instance through an Instance launch role are being used from another account within AWS.
  • Credentials that were created exclusively for an EC2 instance through an Instance launch role are being used from an external IP address.
  • An API was invoked from a known malicious IP address.
  • An API was invoked from an IP address on a custom threat list.
  • An API was invoked from a Tor exit node IP address.

AWS implementation and additional context:

GCP guidance: Use the Event Threat Detection in Google Cloud Security Command Center for certain type of IAM related threat detection such as detection of events where a dormant user-managed service account was granted one or more sensitive IAM roles.

Be aware that Google Identity logs and Google Cloud IAM logs both produce admin activity logs but for the different scope. Google Identity logs are only for operations corresponding to Identity Platform while IAM logs are for operations correspond to IAM for Google Cloud. IAM logs contains log entries of API calls or other actions that modify the configuration or metadata of resources. For example, these logs record when users create VM instances or change Identity and Access Management permissions.

Use the Cloud Identity and IAM reports to alerts on certain suspicious activity patterns. You can use also Policy Intelligence to analyze service accounts activities to identify activities such as service accounts in your project have not been used in the past 90 days.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-3: Enable logging for security investigation

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.2, 8.5, 8.12 AU-3, AU-6, AU-12, SI-4 10.1, 10.2, 10.3

Security principle: Enable logging for your cloud resources to meet the requirements for security incident investigations and security response and compliance purposes.

Azure guidance: Enable logging capability for resources at the different tiers, such as logs for Azure resources, operating systems and applications inside in your VMs and other log types.

Be mindful about different types of logs for security, audit, and other operational logs at the management/control plane and data plane tiers. There are three types of the logs available at the Azure platform:

  • Azure resource log: Logging of operations that are performed within an Azure resource (the data plane). For example, getting a secret from a key vault or making a request to a database. The content of resource logs varies by the Azure service and resource type.
  • Azure activity log: Logging of operations on each Azure resource at the subscription layer, from the outside (the management plane). You can use the Activity Log to determine what, who, and when for any write operations (PUT, POST, DELETE) taken on the resources in your subscription. There is a single Activity log for each Azure subscription.
  • Azure Active Directory logs: Logs of the history of sign-in activity and audit trail of changes made in the Azure Active Directory for a particular tenant.

You can also use Microsoft Defender for Cloud and Azure Policy to enable resource logs and log data collecting on Azure resources.

Azure implementation and additional context:

AWS guidance: Use AWS CloudTrail logging for management events (control plane operations) and data events (data plane operations) and monitor these trails with CloudWatch for automated actions.

The Amazon CloudWatch Logs service allows you to collect and store logs from your resources, applications, and services in near real time. There are three main categories of logs:

  • Vended logs: Logs natively published by AWS services on your behalf. Currently, Amazon VPC Flow Logs and Amazon Route 53 logs are the two supported types. These two logs are enabled by default.
  • Logs published by AWS services: Logs from more than 30 AWS services publish to CloudWatch. They include Amazon API Gateway, AWS Lambda, AWS CloudTrail, and many others. These logs can be enabled directly in the services and CloudWatch.
  • Custom logs: Logs from your own application and on-premises resources. You may need to collect these logs by installing CloudWatch Agent in your operating systems and forward them to CloudWatch.

While many services publish logs only to CloudWatch Logs, some AWS services can publish logs directly to AmazonS3 or Amazon Kinesis Data Firehose where you can use different logging storage and retention policies.

AWS implementation and additional context:

GCP guidance: Enable logging capability for resources at the different tiers, such as logs for Azure resources, operating systems and applications inside in your VMs and other log types.

Be mindful about different types of logs for security, audit, and other operational logs at the management/control plane and data plane tiers. Operations Suite Cloud Logging service collect and aggregate all kind of log events from resource tiers. Four categories of logs are supported in Cloud Logging:

  • Platform logs - logs written by your Google Cloud services.
  • Component logs - similar to platform logs, but they are logs generated by Google-provided software components that run on your systems.
  • Security logs - mainly audit logs that record administrative activities and accesses within your resources.
  • User-written - logs written by custom applications and services
  • Multi-cloud logs and Hybrid-cloud logs - logs from other cloud providers like Microsoft Azure and logs from on-premises infrastructure.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-4: Enable network logging for security investigation

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.2, 8.5, 8.6, 8.7, 13.6 AU-3, AU-6, AU-12, SI-4 10.8

Security principle: Enable logging for your network services to support network-related incident investigations, threat hunting, and security alert generation. The network logs may include logs from network services such as IP filtering, network and application firewall, DNS, flow monitoring and so on.

Azure guidance: Enable and collect network security group (NSG) resource logs, NSG flow logs, Azure Firewall logs, and Web Application Firewall (WAF) logs, and logs from virtual machines via the network traffic data collection agent for security analysis to support incident investigations, and security alert generation. You can send the flow logs to an Azure Monitor Log Analytics workspace and then use Traffic Analytics to provide insights.

Collect DNS query logs to assist in correlating other network data.

Azure implementation and additional context:

AWS guidance: Enable and collect network logs such as VPC Flow Logs, WAF Logs, and Route53 Resolver query logs for security analysis to support incident investigations, and security alert generation. The logs can be exported to CloudWatch for monitoring or an S3 storage bucket for ingesting into the Microsoft Sentinel solution for centralized analytics.

AWS implementation and additional context:

GCP guidance: Most of the network activities logs are available through the VPC Flow Logs which records a sample of network flows send from and received by resources, including instances used as Google Compute VMs, Kubernetes Engine nodes. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.

You can view flow logs in Cloud Logging, and export logs to the destination that Cloud Logging export supports. Flow logs are aggregated by connection from Compute Engine VM’s and exported in real time. By subscribing to Pub/Sub, you can analyze flow logs using real-time streaming APIs.

Note: You can also use Packet Mirroring clones the traffic of specified instances in your Virtual Private Cloud (VPC) network and forwards it for examination. Packet Mirroring captures all traffic and packet data, including payloads and headers.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-5: Centralize security log management and analysis

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.9, 8.11, 13.1 AU-3, AU-6, AU-12, SI-4 N/A

Security principle: Centralize logging storage and analysis to enable correlation across log data. For each log source, ensure that you have assigned a data owner, access guidance, storage location, what tools are used to process and access the data, and data retention requirements.

Use Cloud native SIEM if you don't have an existing SIEM solution for CSPs. or aggregate logs/alerts into your existing SIEM.

Azure guidance: Ensure that you are integrating Azure activity logs into a centralized Log Analytics workspace. Use Azure Monitor to query and perform analytics and create alert rules using the logs aggregated from Azure services, endpoint devices, network resources, and other security systems.

In addition, enable and onboard data to Microsoft Sentinel which provides security information event management (SIEM) and security orchestration automated response (SOAR) capabilities.

Azure implementation and additional context:

AWS guidance: Ensure that you are integrating your AWS logs into a centralized resource for storage and analysis. Use CloudWatch to query and perform analytics, and to create alert rules using the logs aggregated from AWS services, services, endpoint devices, network resources, and other security systems.

In addition, you can aggregate the logs in a S3 storage bucket and onboard the log data to Microsoft Sentinel which provides security information event management (SIEM) and security orchestration automated response (SOAR) capabilities.

AWS implementation and additional context:

GCP guidance: Ensure that you are integrating your GCP logs into a centralized resource (such as Operations Suite Cloud Logging bucket) for storage and analysis. Cloud Logging supports most of the Google Cloud native service logging as well as the third-party applications and on-premise applications. You can use Cloud Logging for query and perform analytics, and to create alert rules using the logs aggregated from GCP services, services, endpoint devices, network resources, and other security systems.

Use Cloud native SIEM if you don’t have an existing SIEM solution for CSP’s, or aggregate logs/alerts into your existing SIEM.

Note: Google provide two log query frontend, Logs Explorer and Log Analytics for query, view, and analyze logs. For troubleshooting and exploring of log data, it is recommended to use Logs Explorer. To generate insights and trends, it is recommended to use Log Analytics.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-6: Configure log storage retention

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.3, 8.10 AU-11 10.5, 10.7

Security principle: Plan your log retention strategy according to your compliance, regulation, and business requirements. Configure the log retention policy at the individual logging services to ensure the logs are archived appropriately.

Azure guidance: Logs such as Azure Activity Logs are retained for 90 days and then deleted. You should create a diagnostic setting and route the logs to another location (such as Azure Monitor Log Analytics workspace, Event Hubs or Azure Storage) based on your needs. This strategy also applies to other resource logs and resources managed by yourself such as logs in the operating systems and applications inside VMs.

You have the log retention option as below:

  • Use Azure Monitor Log Analytics workspace for a log retention period of up to 1 year or per your response team requirements.
  • Use Azure Storage, Data Explorer or Data Lake for long-term and archival storage for greater than 1 year and to meet your security compliance requirements.
  • Use Azure Event Hubs to forward logs to an external resource outside of Azure.

Note: Microsoft Sentinel uses Log Analytics workspace as its backend for log storage. You should consider a long-term storage strategy if you plan to retain SIEM logs for longer time.

Azure implementation and additional context:

AWS guidance: By default, logs are kept indefinitely and never expire in CloudWatch. You can adjust the retention policy for each log group, keeping the indefinite retention, or choosing a retention period between 10 years and one day.

Use Amazon S3 for log archival from CloudWatch and apply object lifecycle management and archival policy to the bucket. You can use Azure Storage for central log archival by transferring the files from Amazon S3 to Azure Storage.

AWS implementation and additional context:

GCP guidance: By default, Operations Suite Cloud Logging retains the logs for 30 days, unless you configure custom retention for the Cloud Logging bucket. Admin Activity audit logs, System Event audit logs, and Access Transparency logs are retained 400 days by default. You can configure Cloud Logging to retain logs between 1 day and 3650 days.

Use Cloud Storage for log archival from Cloud Logging and apply object lifecycle management and archival policy to the bucket. You can use Azure Storage for central log archival by transferring the files from Google Cloud Storage to Azure Storage.

GCP implementation and additional context:

Customer security stakeholders (Learn more):

LT-7: Use approved time synchronization sources

CIS Controls v8 ID(s) NIST SP 800-53 r4 ID(s) PCI-DSS ID(s) v3.2.1
8.4 AU-8 10.4

Security principle: Use approved time synchronization sources for your logging time stamp which include date, time and time zone information.

Azure guidance: Microsoft maintains time sources for most Azure PaaS and SaaS services. For your compute resources operating systems, use a Microsoft default NTP server for time synchronization unless you have a specific requirement. If you need to stand up your own network time protocol (NTP) server, ensure you secure the UDP service port 123.

All logs generated by resources within Azure provide time stamps with the time zone specified by default.

Azure implementation and additional context:

AWS guidance: AWS maintains time sources for most AWS services. For resources or services where the operating system time setting is configured, use AWS default Amazon Time Sync Service for time synchronization unless you have a specific requirement. If you need to stand up your own network time protocol (NTP) server, ensure you secure the UDP service port 123.

All logs generated by resources within AWS provide time stamps with the time zone specified by default.

AWS implementation and additional context:

GCP guidance: Google Cloud maintains time sources for most Google Cloud PaaS and SaaS services. For your compute resources operating systems, use a Google Cloud default NTP server for the time synchronization unless you have a specific requirement. If you need to stand up your own network time protocol (NTP) server, ensure to secure the UDP service port 123.

Note: it is recommended that you do not use external NTP sources with Compute Engine virtual machines but use the internal NTP server provided by Google.

GCP implementation and additional context:

Customer security stakeholders (Learn more):