ADF Inline dataset parameterization of folderpath

Question

Using ADF dataflow to copy databricks delta files to sql database. Under Source Options of dataflow , file system and folder path are not able to be parameterized. My objective is get the filesystem and folderpath value from previous activity.

Answer 1

Hello @Prakash14 ,

Thanks for the question and using MS Q&A platform.

In order to parameterize the fileSystem and folderPath of your delta source, you will have to follow the below procedure of steps:

1. First create a dataflow parameters for both FileSystem and FolderPath as shown below

1. Then go to your source options section, in Folder path properties, select the data flow parameters for fileSystem and folderPath you have created in step 1 using the expression builder.

3. Then go your pipeline level settings, and click on data flow activity, then you will see the dataflow parameters below which you have configured in earlier steps. Next check box the Expression field and then under value property, click on pipeline expression and add the dynamic expression to pass the output of your previous activity from which you want to pass the values. Below is a sample GIF for the same.

GIF:

Hope this will help. Please let us know if any further queries.

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Answer 2

df_container is the ADLS gen2 container name.
df_folderpath is the ADLS gen2 folder path

df_container : source , df_folderpath : input/data/

Inside the source container - source/input/data/ there will be multiple directories and delta format files(databricks) will be there under each partition .

date=2017-01-01/abc.parquet
date=2017-03-01/xyz.parquet
date=2017-04-01/pqr.parquet