question

SateeshBattu-5298 avatar image
0 Votes"
SateeshBattu-5298 asked SateeshBattu-5298 commented

Programetically getting data flow source "source options -> Folder path" for inline source type

Hi ,

In a data flow i have defined online source type . In the source options, defined the ADLS delta file location

Now i would like to programmatically ( any possible programming language using azure - cli/JAVA/Python/powershell/APIs ) get the value of the source option folder path dynamically from run time metadata.
If it is a dataset type, i can use "az datafactory dataset show " command in cli and get the schema table information.
How to get the same in case it is inline ADLS delta file.


201421-image.png

201260-image.png

The path highlighted in the second image want to get from activities metadata. or any other command in azure. Please help


azure-data-factory
image.png (42.2 KiB)
image.png (44.6 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ SateeshBattu-5298 . We have had a bit of a backlog of questions to work through, so I didn't see you til now.

Okay, so if I understand correctly, you want to not set it at runtime, but fetch afterwards what the expression evaluated to.

I don't think there is a command explicitly for this, so I will have to dig a bit and see what sideways approaches will work.

1 Vote 1 ·

@MartinJaffer-MSFT

is this same in case of COPY activity where source and destination path / table information is an expression, can we get the calculated value of the source / destination dataset in the activity output of any where .

My copy activity is using variables / pipeline parameters or current time in the expression for source/target ( sink ) object of the COPY activity.

202691-image.png


202567-image.png


0 Votes 0 ·
image.png (47.9 KiB)
image.png (49.0 KiB)

This is bit urgent, request your response ASAP

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered MartinJaffer-MSFT commented

@SateeshBattu-5298
I notice that your path is concatenating parameters. If these parameters are passed into the Data Flow from the pipeline, then one solution is to simply reconstruct the path from the same parameters. I think the evaluations inside a dataflow are not exposed, so as to better keep information privacy.

The Data Flow script itself will tell you the default value of a parameter, and the expressions used. properties.scriptLines


The activity runs will tell you what the input parameter (to the Data Flow) values. value[n].input.dataflow.parameters[parameterName]

If your parameter values are being obtained from another source, things can get more complicated.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@MartinJaffer-MSFT

We are on the same line, I accept this answer.

If you can give some more details about the format of the properties.scriptLinesfrom get data flows or any indication to parse parameters and expression used in the folder path from properties.scriptLines that will be more useful for me.

0 Votes 0 ·

The scriptlines are strings inside, which will need some custom work to parse, not a simple JSON member. That is why I didn't go into detail there.
I could put in a sample, but I'm sure you will see enough of those while developing your solution.
@SateeshBattu-5298

0 Votes 0 ·