Azure Data Factory Question

Question

Azure Data Factory Question

Hi, I want to extract data from an API, and I'm confused about which activity in ADF I should use. Should I go with COPY or WEB activity?

Can anyone please help?

Answer 1

Hi Ayaan Shaikh,

To showcase you how to do extract data from an API I would like to share with you this terraform template that deploy adf and create the pipeline that extract the API response:

resource "azurerm_resource_group" "rg_experiment_adf" {
  name     = "rg-experiment-adf"
  location = "eastus2"
}

resource "azurerm_data_factory" "adf_experiment_01" {
  name                = "adf-experiment-01"
  location            = azurerm_resource_group.rg_experiment_adf.location
  resource_group_name = azurerm_resource_group.rg_experiment_adf.name
}

resource "azurerm_data_factory_pipeline" "web_scrapper_pipeline" {
  name            = "web-scrapper-pipeline"
  data_factory_id = azurerm_data_factory.adf_experiment_01.id

  variables = {
    "response" = "initial_value"
  }

  activities_json = <<JSON
[
  {
    "name": "GetApiData",
    "type": "WebActivity",
    "typeProperties": {
      "url": "https://pokeapi.co/api/v2/pokemon",
      "method": "GET"
    }
  }
]
JSON
}

So this pipeline show the web activity and get the response :

After that you can decide what do you want to do with the output, for example transform, copy or others.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

Regards,

Luis

Answer 2

Great question! The choice between COPY and WEB activity in Azure Data Factory (ADF) depends on your specific requirements:

• COPY Activity: Use this if your primary goal is to move data from an API to a data store with minimal data transformation. COPY activity is optimized for data transfer and can handle various data formats and sources.
• WEB Activity: Use this if you need more advanced API interactions or want to integrate with other activities in a pipeline. WEB activity is better suited for scenarios where you need to perform complex operations or interact with REST APIs that require custom headers, authentication, or other advanced settings.

In summary, if you're simply transferring data with little to no transformation, COPY activity is the way to go. If you need more control and customization, WEB activity is the better choice.

https://www.journeyteam.com/resources/blog/copy-activity-vs-web-activity/

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful **