Mapping Address Values to Relative URLs in Azure Data Factory

Question

In Azure Data Factory, how can I map address values obtained from a source dataset to their corresponding relative URLs stored in a lookup dataset? Could you provide a step-by-step guide on configuring a lookup activity to achieve this mapping within a pipeline? I used a lookup activity, I am not sure how to configure the parameters, so the relative url can be read dynamically from the SINK REST relative url option.

Answer 1

First, you need to create a dataset for your source data, which contains the address values, and properly configured (since you didn't mention your source).

Second, you will have to create a dataset for the lookup table that contains address values and their corresponding relative URLs.

Then, you need to configure your pipeline :

• Drag a Lookup activity onto the pipeline canvas.
• Configure the Lookup activity:
  • Settings:
    • Select the Lookup Dataset you created earlier.
    • Choose First row only if you expect a single match or All if multiple matches are expected.
  • Query: It is optional and you can write a query to filter the lookup dataset based on the address values if necessary.

Now, you need to Add ForEach Activity:

• Drag a ForEach activity onto the pipeline canvas.
• Set the Items property of the ForEach activity to the output of the Lookup activity.
• For example: @activity('Lookup1').output.value

And then, you need to add Data Flow Activity:

• Inside the ForEach activity, add a Data Flow activity.
• Configure the Data Flow activity:
  • Source:
    • Select the Source Dataset.
    • Filter the data based on the current item in the ForEach loop.
  • Sink:
    • Configure the Sink to be the REST API or another target where the relative URL is required.
    • Set the relative URL dynamically using the output from the Lookup activity.

Now, you need to configure the parameters and the dynamic content.

You need to define a parameter in the Data Flow for the relative URL (for example relativeUrl)

Then in the Sink settings, map the parameter to the relative URL field.

Use dynamic content to set the parameter value:

@activity('Lookup1').output.value[0]['RelativeURL']

Finally, if you are using a REST API Sink, configure it to use the relativeUrl parameter in the URL.

Example: /api/resource/@{pipeline().parameters.relativeUrl}

Links to guide you :

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-lookup

https://www.metadapi.com/Blog/dynamically-invoking-rest-api-with-data-factory