How to copy data from google secops platform using ADF

Question

Hi there,

How to copy data from google secops platform using ADF

Thanks & Best regards,

zmsoft

Answer 1

Hi @zmsoft

To copy data from Google SecOps (Chronicle) into Azure Data Factory (ADF), there is currently no native Google SecOps connector available in ADF.

The recommended approach depends on how the SecOps data is exposed.

If the Google SecOps data is exported to Google Cloud Storage (GCS), Azure Data Factory can ingest the data using the Google Cloud Storage connector.

Prerequisite: Ensure that Google SecOps is configured to export data to a customer-managed GCS bucket (for example using Google SecOps Data Export APIs). Without exporting the data first, ADF cannot directly access Google SecOps data.

Steps:

1. Prepare GCS access

• Enable interoperability access for the GCS bucket
• Create a service account with appropriate storage permissions
• Generate HMAC credentials (Access Key ID and Secret Access Key)

1. Create Linked Service in ADF

• Create a Google Cloud Storage linked service
• Configure:
  • Service URL: https://storage.googleapis.com
    • HMAC credentials
      • Integration Runtime (Azure IR or Self-hosted IR if required)
      • Validate the connection

1. Create Dataset

• Configure the bucket and folder/file path
• Use wildcard patterns if required

1. Create Copy Pipeline

• Source: Google Cloud Storage dataset
• Sink: Azure Blob Storage / ADLS Gen2 / Azure SQL / Synapse
• Configure mappings and transformations if needed

1. Execute and Monitor

• Trigger the pipeline
• Monitor execution and performance in ADF

Alternative Approach: If GCS export is not available, ADF can use REST connector integration with Google SecOps APIs. This approach requires handling authentication, pagination, and API limitations and is generally less suitable for large-scale ingestion scenarios.

Recommendation: For production scenarios, exporting Google SecOps data to GCS and ingesting through ADF is typically the most scalable and operationally manageable approach.

Reference:
https://learn.microsoft.com/en-us/azure/data-factory/connector-google-cloud-storage?tabs=data-factory
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
https://learn.microsoft.com/en-us/azure/data-factory/connector-overview

Please let us know if you have any questions or concerns.

Answer 2

Hey zmsoft,

It sounds like you want to bring data out of a Google SecOps-hosted bucket into Azure via Data Factory. The easiest path is to treat your SecOps data as files in Google Cloud Storage (GCS) and use ADF’s Google Cloud Storage connector (S3-API compatible). Here’s a high-level recipe:

1. Prepare your GCS side • Enable interoperability on the GCS bucket where your SecOps data lives. • Create a Google service account with at least storage.objects.get and storage.objects.list roles (and storage.buckets.list if you want UI browsing). • Generate an HMAC access key (AccessKeyID + SecretAccessKey).
2. Create a GCS Linked Service in ADF • In the Azure portal, open your Data Factory → Manage → Linked services → New. • Search “Google Cloud Storage (S3 API)” and select it. • Fill in: – serviceUrl: https://storage.googleapis.com – accessKeyId / secretAccessKey (or reference Key Vault secrets) – choose your Integration Runtime (Azure IR or Self-hosted IR if needed) • Test the connection and Create.
3. Build your source dataset • In ADF Authoring, add a new Dataset → choose Google Cloud Storage. • Point to your bucket and folder or file path. You can use wildcards (e.g. /logs/**/.json).
4. Build your destination (sink) dataset • This could be Blob Storage, ADLS Gen2, SQL Database, Synapse, etc. • Configure the linked service and file/folder/table details accordingly.
5. Author the Copy Activity • In a pipeline, drop in a Copy Data activity. • Source: point to your GCS dataset. • Sink: point to your destination dataset. • (Optional) Adjust schema mappings, enable staged copy, filter rows, etc.
6. Run & Monitor • Trigger the pipeline manually or schedule it. • Check Monitoring to see throughput, errors, performance stats.

Pro tip: If you just want a quick wizard to scaffold everything—including integration runtime, linked services, datasets and pipeline—launch the Copy Data tool in the ADF UI (aka Copy Wizard). It’ll walk you through source, sink, mappings and scheduling without hand-coding JSON.

Hope this gives you a clear path forward—let me know if you need more details!

Reference list

• Copy Data Tool (Wizard)

https://docs.microsoft.com/azure/data-factory/copy-data-tool

• ADF Copy Activity overview

https://docs.microsoft.com/azure/data-factory/copy-activity-overview

• Supported connectors in Azure Data Factory

https://docs.microsoft.com/azure/data-factory/connector-overview

• Copy data from Google Cloud Storage using ADF

https://docs.microsoft.com/azure/data-factory/connector-google-cloud-storage?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#prerequisites

–and–

https://docs.microsoft.com/azure/data-factory/connector-google-cloud-storage?wt.mc_id=knowledgesearch_inproduct_azure-cxp-community-insider#create-a-linked-service-to-google-cloud-storage-using-ui

Answer 3

Google SecOps is not listed as a supported source connector in the provided context. However, Azure Data Factory can copy data from several Google services that might be used as export targets for SecOps data.

If Google SecOps can export or stream its data into one of the following, then Azure Data Factory can pick it up from there:

1. Google Ads → ADF
   If SecOps data is related to Google Ads and is exposed via the Google Ads API, use the built-in Google Ads connector:
   • Create a Google Ads linked service in ADF with type GoogleAdWords, specifying:
     • googleAdsApiVersion
     • clientCustomerID
     • developerToken
   • Create a dataset with type GoogleAdWordsObject.
   • In a Copy activity, set the source type to GoogleAdWordsSource and use a GAQL query (for example: SELECT campaign.id FROM campaign).
2. Google Cloud Storage → ADF
   If Google SecOps can export logs or events as files (CSV, JSON, etc.) into Google Cloud Storage (GCS), then use the Google Cloud Storage connector:
   • Configure a GCS linked service in ADF.
   • Create a file-based dataset pointing to the GCS bucket/path where SecOps exports data.
   • Use Copy activity (or Mapping Data Flow) with GCS as source and your target (for example, Azure Data Lake Storage, Azure SQL, Synapse) as sink.
   • The connector supports copying files as-is or parsing supported file formats.

In practice, the pattern is:

1. Configure Google SecOps to export data into either Google Ads (if applicable) or Google Cloud Storage (more general/log-based scenario).
2. Use the corresponding ADF connector (Google Ads or Google Cloud Storage) as the source in a Copy activity.
3. Write the data to the desired Azure sink (Data Lake, SQL, Synapse, etc.).

If Google SecOps cannot export to Google Ads or GCS, a custom integration (for example, via an intermediate service or API-based ingestion) would be required, which is not covered in the provided context.

---

References:

• Copy data from Google Ads using Azure Data Factory or Synapse Analytics
• Copy data from Google Cloud Storage using Azure Data Factory or Synapse Analytics