Ingest Google Cloud Platform log data into Microsoft Sentinel

Organizations are increasingly moving to multicloud architectures, whether by design or due to ongoing requirements. A growing number of these organizations use applications and store data on multiple public clouds, including the Google Cloud Platform (GCP).

This article describes how to ingest GCP data into Microsoft Sentinel to get full security coverage and analyze and detect attacks in your multicloud environment.

With the GCP Pub/Sub connectors, based on our Codeless Connector Platform (CCP), you can ingest logs from your GCP environment using the GCP Pub/Sub capability:

• The Google Cloud Platform (GCP) Pub/Sub Audit Logs connector collects audit trails of access to GCP resources. Analysts can monitor these logs to track resource access attempts and detect potential threats across the GCP environment.

• The Google Cloud Platform (GCP) Security Command Center connector collects findings from Google Security Command Center, a robust security and risk management platform for Google Cloud. Analysts can view these findings to gain insights into the organization's security posture, including asset inventory and discovery, detections of vulnerabilities and threats, and risk mitigation and remediation.

Important

The GCP Pub/Sub connectors are currently in PREVIEW. The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Prerequisites

Before you begin, verify that you have the following:

• The Microsoft Sentinel solution is enabled.
• A defined Microsoft Sentinel workspace exists.
• A GCP environment exists and contains resources producing one of the following log type you want to ingest:
  • GCP audit logs
  • Google Security Command Center findings
• Your Azure user has the Microsoft Sentinel Contributor role.
• Your GCP user has access to create and edit resources in the GCP project.
• The GCP Identity and Access Management (IAM) API and the GCP Cloud Resource Manager API are both enabled.

Set up GCP environment

There are two things you need to set up in your GCP environment:

1. Set up Microsoft Sentinel authentication in GCP by creating the following resources in the GCP IAM service:

   • Workload identity pool
   • Workload identity provider
   • Service account
   • Role

2. Set up log collection in GCP and ingestion into Microsoft Sentinel by creating the following resources in the GCP Pub/Sub service:

   • Topic
   • Subscription for the topic

You can set up the environment in one of two ways:

• Create GCP resources via the Terraform API: Terraform provides APIs for resource creation and for Identity and Access Management (see Prerequisites). Microsoft Sentinel provides Terraform scripts that issue the necessary commands to the APIs.

• Set up GCP environment manually, creating the resources yourself in the GCP console.

  Note

  There is no Terraform script available for creating GCP Pub/Sub resources for log collection from Security Command Center. You must create these resources manually. You can still use the Terraform script to create the GCP IAM resources for authentication.

  Important

  If you're creating resources manually, you must create all the authentication (IAM) resources in the same GCP project, otherwise it won't work. (Pub/Sub resources can be in a different project.)

GCP Authentication Setup

Terraform API Setup

1. Open GCP Cloud Shell.

2. Select the project you want to work with, by typing the following command in the editor:

   gcloud config set project {projectId}  
   

3. Copy the Terraform authentication script provided by Microsoft Sentinel from the Sentinel GitHub repository into your GCP Cloud Shell environment.

   1. Open the Terraform GCPInitialAuthenticationSetup script file and copy its contents.

      Note

      For ingesting GCP data into an Azure Government cloud, use this authentication setup script instead.

   2. Create a directory in your Cloud Shell environment, enter it, and create a new blank file.

      mkdir {directory-name} && cd {directory-name} && touch initauth.tf
      

   3. Open initauth.tf in the Cloud Shell editor and paste the contents of the script file into it.

4. Initialize Terraform in the directory you created by typing the following command in the terminal:

   terraform init 
   

5. When you receive the confirmation message that Terraform was initialized, run the script by typing the following command in the terminal:

   terraform apply 
   

6. When the script prompts for your Microsoft tenant ID, copy and paste it into the terminal.

   Note

   You can find and copy your tenant ID on the GCP Pub/Sub Audit Logs connector page in the Microsoft Sentinel portal, or in the Portal settings screen (accessible anywhere in the Azure portal by selecting the gear icon along the top of the screen), in the Directory ID column.

7. When asked if a workload Identity Pool has already been created for Azure, answer yes or no accordingly.

8. When asked if you want to create the resources listed, type yes.

When the output from the script is displayed, save the resources parameters for later use.

Manual setup

Create and configure the following items in the Google Cloud Platform Identity and Access Management (IAM) service.

Create a custom role

1. Follow the instructions in the Google Cloud documentation to create a role. Per those instructions, create a custom role from scratch.

2. Name the role so it's recognizable as a Sentinel custom role.

3. Fill in the relevant details and add permissions as needed:

   • pubsub.subscriptions.consume
   • pubsub.subscriptions.get

   You can filter the list of available permissions by roles. Select the Pub/Sub Subscriber and Pub/Sub Viewer roles to filter the list.

For more information about creating roles in Google Cloud Platform, see Create and manage custom roles in the Google Cloud documentation.

Create a service account

1. Follow the instructions in the Google Cloud documentation to create a service account.

2. Name the service account so it's recognizable as a Sentinel service account.

3. Assign the role you created in the previous section to the service account.

For more information about service accounts in Google Cloud Platform, see Service accounts overview in the Google Cloud documentation.

Create the workload identity pool and provider

1. Follow the instructions in the Google Cloud documentation to create the workload identity pool and provider.

2. For the Name and Pool ID, enter your Azure Tenant ID, with the dashes removed.

   Note

   You can find and copy your tenant ID on the Portal settings screen, in the Directory ID column. The portal settings screen is accessible anywhere in the Azure portal by selecting the gear icon along the top of the screen.

3. Add an identity provider to the pool. Choose Open ID Connect (OIDC) as the provider type.

4. Name the identity provider so it's recognizable for its purpose.

5. Enter the following values in the provider settings (these aren't samples—use these actual values):

   • Issuer (URL): https://sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d
   • Audience: the application ID URI: api://2041288c-b303-4ca0-9076-9612db3beeb2
   • Attribute mapping: google.subject=assertion.sub

   Note

   To set up the connector to send logs from GCP to the Azure Government cloud, use the following alternate values for the provider settings instead of those above:

   • Issuer (URL): https://sts.windows.net/cab8a31a-1906-4287-a0d8-4eef66b95f6e
   • Audience: api://e9885b54-fac0-4cd6-959f-a72066026929

For more information about workload identity federation in Google Cloud Platform, see Workload identity federation in the Google Cloud documentation.

Grant the identity pool access to the service account

1. Locate and select the service account you created earlier.

2. Locate the permissions configuration of the service account.

3. Grant access to the principal that represents the workload identity pool and provider that you created in the previous step.

   • Use the following format for the principal name:

     principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{WORKLOAD_IDENTITY_POOL_ID}/subject/{WORKLOAD_IDENTITY_PROVIDER_ID}
     

   • Assign the Workload Identity User role and save the configuration.

For more information about granting access in Google Cloud Platform, see Manage access to projects, folders, and organizations in the Google Cloud documentation.

GCP Audit Logs Setup

The instructions in this section are for using the Microsoft Sentinel GCP Pub/Sub Audit Logs connector.

See the instructions in the next section for using the Microsoft Sentinel GCP Pub/Sub Security Command Center connector.

Terraform API Setup

1. Copy the Terraform audit log setup script provided by Microsoft Sentinel from the Sentinel GitHub repository into a different folder in your GCP Cloud Shell environment.

   1. Open the Terraform GCPAuditLogsSetup script file and copy its contents.

      Note

      For ingesting GCP data into an Azure Government cloud, use this audit log setup script instead.

   2. Create another directory in your Cloud Shell environment, enter it, and create a new blank file.

      mkdir {other-directory-name} && cd {other-directory-name} && touch auditlog.tf
      

   3. Open auditlog.tf in the Cloud Shell editor and paste the contents of the script file into it.

2. Initialize Terraform in the new directory by typing the following command in the terminal:

   terraform init 
   

3. When you receive the confirmation message that Terraform was initialized, run the script by typing the following command in the terminal:

   terraform apply 
   

   To ingest logs from an entire organization using a single Pub/Sub, type:

   terraform apply -var="organization-id= {organizationId} "
   

4. When asked if you want to create the resources listed, type yes.

When the output from the script is displayed, save the resources parameters for later use.

Wait five minutes before moving to the next step.

Manual setup

Create a publishing topic

Use the Google Cloud Platform Pub/Sub service to set up export of audit logs.

Follow the instructions in the Google Cloud documentation to create a topic for publishing logs to.

• Choose a Topic ID that reflects the purpose of log collection for export to Microsoft Sentinel.
• Add a default subscription.
• Use a Google-managed encryption key for encryption.

Create a log sink

Use the Google Cloud Platform Log Router service to set up collection of audit logs.

To collect logs for resources in the current project only:

1. Verify that your project is selected in the project selector.

2. Follow the instructions in the Google Cloud documentation to set up a sink for collecting logs.

   • Choose a Name that reflects the purpose of log collection for export to Microsoft Sentinel.
   • Select "Cloud Pub/Sub topic" as the destination type, and choose the topic you created in the previous step.

To collect logs for resources throughout the entire organization:

1. Select your organization in the project selector.

2. Follow the instructions in the Google Cloud documentation to set up a sink for collecting logs.

   • Choose a Name that reflects the purpose of log collection for export to Microsoft Sentinel.
   • Select "Cloud Pub/Sub topic" as the destination type, and choose the default "Use a Cloud Pub/Sub topic in a project".
   • Enter the destination in the following format: pubsub.googleapis.com/projects/{PROJECT_ID}/topics/{TOPIC_ID}.

3. Under Choose logs to include in the sink, select Include logs ingested by this organization and all child resources.

Verify that GCP can receive incoming messages

1. In the GCP Pub/Sub console, navigate to Subscriptions.

2. Select Messages, and select PULL to initiate a manual pull.

3. Check the incoming messages.

If you're also setting up the GCP Pub/Sub Security Command Center connector, continue with the next section.

Otherwise, skip to Set up the GCP Pub/Sub connector in Microsoft Sentinel.

GCP Security Command Center setup

The instructions in this section are for using the Microsoft Sentinel GCP Pub/Sub Security Command Center connector.

See the instructions in the previous section for using the Microsoft Sentinel GCP Pub/Sub Audit Logs connector.

Configure continuous export of findings

Follow the instructions in the Google Cloud documentation to configure Pub/Sub exports of future SCC findings to the GCP Pub/Sub service.

1. When asked to select a project for your export, select a project you created for this purpose, or create a new project.

2. When asked to select a Pub/Sub topic where you want to export your findings, follow the instructions above to create a new topic.

Set up the GCP Pub/Sub connector in Microsoft Sentinel

GCP Audit Logs

1. Open the Azure portal and navigate to the Microsoft Sentinel service.

2. In the Content hub, in the search bar, type Google Cloud Platform Audit Logs.

3. Install the Google Cloud Platform Audit Logs solution.

4. Select Data connectors, and in the search bar, type GCP Pub/Sub Audit Logs.

5. Select the GCP Pub/Sub Audit Logs (Preview) connector.

6. In the details pane, select Open connector page.

7. In the Configuration area, select Add new collector.

   

8. In the Connect a new collector panel, type the resource parameters you created when you created the GCP resources.

   

9. Make sure that the values in all the fields match their counterparts in your GCP project (the values in the screenshot are samples, not literals), and select Connect.

Google Security Command Center

1. Open the Azure portal and navigate to the Microsoft Sentinel service.

2. In the Content hub, in the search bar, type Google Security Command Center.

3. Install the Google Security Command Center solution.

4. Select Data connectors, and in the search bar, type Google Security Command Center.

5. Select the Google Security Command Center (Preview) connector.

6. In the details pane, select Open connector page.

7. In the Configuration area, select Add new collector.

   

8. In the Connect a new collector panel, type the resource parameters you created when you created the GCP resources.

   

9. Make sure that the values in all the fields match their counterparts in your GCP project (the values in the screenshot are samples, not literals), and select Connect.

Verify that the GCP data is in the Microsoft Sentinel environment

1. To ensure that the GCP logs were successfully ingested into Microsoft Sentinel, run the following query 30 minutes after you finish to set up the connector.

   GCP Audit Logs

   GCPAuditLogs 
   | take 10 
   

   Google Security Command Center

   GoogleSCC 
   | take 10 
   

2. Enable the health feature for data connectors.

Next steps

In this article, you learned how to ingest GCP data into Microsoft Sentinel using the GCP Pub/Sub connectors. To learn more about Microsoft Sentinel, see the following articles:

• Learn how to get visibility into your data, and potential threats.
• Get started detecting threats with Microsoft Sentinel.
• Use workbooks to monitor your data.