An csv file with 200 columns in adls. fetch only list of columns from config table. config table is in Azure SQL. config table has only 10 col names

19B01A04F3 P 0 Reputation points
2024-05-03T07:59:15.38+00:00

An csv file with 200 columns in adls. fetch only the list of columns from the config table. config table is in Azure SQL.config table has only 10 column names. how to create pipeline with this scenario in Azure datafactory?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,803 questions
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 33,241 Reputation points Microsoft Employee
    2024-05-03T19:27:13.7+00:00

    Hi 19B01A04F3 P ,

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    As per my understanding you are trying to fetch only few selected columns of csv, which are mentioned in a config table using ADF pipeline. Please let me know if my understanding is incorrect.

    Kindly try any one of the below approaches to achieve the requirement:

    • You can use mapping section of copy activity to customize the schema as per the requirement and deselect the unnecessary ones. If not manually, then you can create the json having the needed columns and use it for dynamic schema mapping. Here is more details on that: Dynamic Column mapping in Copy Activity in Azure Data Factory
    • Copy the csv data into SQL table and write the desired select query removing the unwanted columns.

    Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.