Create job_execution_id and get pipeline_run id in the dataflow

Question

Create job_execution_id and get pipeline_run id in the dataflow

Aditya Singh 160

I am creating a file in the dataflow and want to create a column for job_execution id so that every times the job run I get a unique id for the file and also want to get the pipeline_run_id on every execution.

Sample job

job_execution_id and pipeline_id User's image

Richard Swinbank 527 Reputation points

2024-03-12T12:35:53.82+00:00
Is your question about an ADF data flow?

I'm not clear what you mean by job_execution_id

You can access the pipeline run ID using the pipeline expression @pipeline.RunId and pass it into the data flow as a data parameter.
ADITYA SINGH 0 Reputation points

2024-03-13T02:13:26.74+00:00

Thanks I am able to get the pipeline id and the pipeline_ecution_id is a column which I want to create so the logic is everytime I run the dataflow a unique id should be generated E.g. first time If I run the 1 second time 2 and so on
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator

2024-03-15T04:29:44.2333333+00:00

ADITYA SINGH We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Richard Swinbank 527 Reputation points

2024-03-12T12:35:53.82+00:00

Is your question about an ADF data flow?

I'm not clear what you mean by job_execution_id

You can access the pipeline run ID using the pipeline expression @pipeline.RunId and pass it into the data flow as a data parameter.
ADITYA SINGH 0 Reputation points

2024-03-13T02:13:26.74+00:00

Thanks I am able to get the pipeline id and the pipeline_ecution_id is a column which I want to create so the logic is everytime I run the dataflow a unique id should be generated E.g. first time If I run the 1 second time 2 and so on
Smaran Thoomu 32,535 Reputation points Microsoft External Staff Moderator

2024-03-15T04:29:44.2333333+00:00

ADITYA SINGH We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Are you running the dataflow multiple times from a single pipeline run? If so, there are a few ways to generate your own unique ID:

you can use the Lookup activity to create a log record in a database table with an IDENTITY column, then return the generated identity to ADF. The generated identity can be passed to the dataflow as a parameter.
you can use the guid() pipeline expression function to generate a GUID in the pipeline iteself, then pass the generated value to the dataflow

If you are only running the dataflow once per pipeline run, I would suggest simply using the pipeline run ID as it will correspond one-to-one with the dataflow execution.

Share via

Create job_execution_id and get pipeline_run id in the dataflow

1 answer

Your answer