Edit

Stream Google Cloud Platform logs into Microsoft Sentinel

  • Article

Organizations are increasingly moving to multicloud architectures, whether by design or due to ongoing requirements. A growing number of these organizations use applications and store data on multiple public clouds, including the Google Cloud Platform (GCP).

This article describes how to ingest GCP data into Microsoft Sentinel to get full security coverage and analyze and detect attacks in your multicloud environment.

With the GCP Pub/Sub Audit Logs connector, based on our Codeless Connector Platform (CCP), you can ingest logs from your GCP environment using the GCP Pub/Sub capability.

Important

The GCP Pub/Sub Audit Logs connector is currently in PREVIEW. The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

Once you ingest the GCP data, you can view the details of three types of audit logs:

  • Admin activity logs
  • Data access logs
  • Access transparency logs

With these, Google's Cloud Audit Logs records a trail that practitioners can use to monitor access and detect potential threats across GCP resources.

Prerequisites

Before you begin, verify that you have:

  • The Microsoft Sentinel solution enabled.
  • A defined Microsoft Sentinel workspace.
  • A GCP environment collecting GCP audit logs.
  • The Microsoft Sentinel Contributor role.
  • Access to edit and create resources in the GCP project.

Set up GCP environment

You can set up the GCP environment in one of two ways:

Create GCP resources via the Terraform API

  1. Open GCP Cloud Shell.

  2. Open the editor and type:

    gcloud config set project {projectId}

  3. In the next window, select Authorize.

  4. Copy the Terraform GCPInitialAuthenticationSetup script, paste the script to a new file, and save it as a .tf file.

  5. In the editor, type:

    terraform init

  6. Type:

    terraform apply

  7. Type your Microsoft tenant ID. Learn how to find your tenant ID.

  8. When asked if a workload Identity Pool has already been created for Azure, type yes or no.

  9. When asked if you want to create the resources listed, type yes.

  10. Save the resources parameters for later use.

  11. In a new folder, copy the Terraform GCPAuditLogsSetup script into a new file, and save it as a .tf file:

    cd {foldername}

  12. In the editor, type:

    terraform init

  13. Type:

    terraform apply

    To ingest logs from an entire organization using a single Pub/Sub, type:

    terraform apply -var="organization-id= {organizationId} "

  14. Type yes.

  15. Save the resource parameters for later use.

  16. Wait five minutes before moving to the next step.

Set up the GCP Pub/Sub connector in Microsoft Sentinel

  1. Open the Azure portal and navigate to the Microsoft Sentinel service.
  2. In the Content hub, in the search bar, type Google Cloud Platform Audit Logs.
  3. Install the Google Cloud Platform Audit Logs solution.
  4. Select Data connectors, and in the search bar, type GCP Pub/Sub Audit Logs.
  5. Select the GCP Pub/Sub Audit Logs (Preview) connector.
  6. Below the connector description, select Open connector page.
  7. In the Configuration area, select Add new.
  8. Type the resource parameters you created when you created the GCP resources. Make sure that the Data Collection Endpoint Name and the Data Collection Rule Name begin with Microsoft-Sentinel- and select Connect.

Verify that the GCP data is in the Microsoft Sentinel environment

  1. To ensure that the GCP logs were successfully ingested into Microsoft Sentinel, run the following query 30 minutes after you finish to set up the connector.

    GCPAuditLogs 
| take 10

  2. Enable the health feature for data connectors.

Set up the GCP environment manually via the GCP portal

This section shows you how to set up the GCP environment manually. Alternatively, you can set up the environment via the Terraform API. If you already set up the environment via the API, skip this section.

Create the role

  1. In the GCP console, navigate to IAM & Admin.

  2. Select Roles and select Create role.

  3. Fill in the relevant details and add permissions as needed.

  4. Filter the permissions by the Pub/Sub Subscriber and Pub/Sub Viewer roles, and select pubsub.subscriptions.consume and pubsub.subscriptions.get permissions.

  5. To confirm, select ADD.

    Screenshot of adding permissions when adding a GCP role.

  6. To create the role, select Create.

Create the service account

  1. In the GCP Console, navigate to Service Accounts, and select Create Service Account.
  2. Fill in the relevant details and select Create and continue.
  3. Select the role you created previously, and select Done to create the service account.

Create the workload identity federation

  1. In the GCP Console, navigate to Workload Identity Federation.

  2. If it's your first time using this feature, select Get started. Otherwise, select Create pool.

  3. Fill in the required details, and make sure that the Tenant ID and Tenant name is the TenantID without dashes.

    Note

    To find the tenant ID, in the Azure portal, navigate to All Services > Azure Active Directory > Overview and copy the TenantID.

  4. Make sure that Enable pool is selected.

    Screenshot of creating the identity pool as part of creating the GCP workload identity federation.

  5. To add a provider to the pool:

    • Select OIDC

    • Type the Issuer (URL): \https://sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d

    • Next to Audiences, select Allowed audiences, and next to Audience 1, type: api://2041288c-b303-4ca0-9076-9612db3beeb2.

      Screenshot of adding the provider to the pool when creating the GCP workload identity federation.

      Screenshot of adding the provider pool audiences when creating the GCP workload identity federation.

Configure the provider attributes

  1. Under OIDC 1, select assertion.sub.

    Screenshot of configuring the GCP provider attributes.

  2. Select Continue and Save.

  3. In the Workload Identity Pools main page, select the created pool.

  4. Select Grant access, select the service account you created previously, and select All identities in the pool as the principals.

    Screenshot of granting access to the GCP service account.

  5. Confirm that the connected service account is displayed.

    Screenshot of viewing the connected GCP service accounts.

Create a topic

  1. In the GCP console, navigate to Topics.
  2. Create a new topic and select a Topic ID.
  3. Select Add default subscription and under Encryption, select Google-managed encryption key.

Create a sink

  1. In the GCP console, navigate to Log Router.

  2. Select Create sink and fill in the relevant details.

  3. Under Sink destination, select Cloud Pub/Sub topic and select the topic you created previously.

    Screenshot of defining the GCP sink destination.

  4. If needed, filter the logs by selecting specific logs to include. Otherwise, all logs are sent.

  5. Select Create sink.

Note

To ingest logs for the entire organization:

  1. Select the organization under Project.
  2. Repeat steps 2-4, and under Choose logs to include in the sink in the Log Router section, select Include logs ingested by this organization and all child resources.

Screenshot of choosing which GCP logs to include in the sink.

Verify that GCP can receive incoming messages

  1. In the GCP console, navigate to Subscriptions.
  2. Select Messages, and select PULL to initiate a manual pull.
  3. Check the incoming messages.

Next steps

In this article, you learned how to ingest GCP data into Microsoft Sentinel using the GCP Pub/Sub Audit Logs connector. To learn more about Microsoft Sentinel, see the following articles: