Saving the output from a execute Pipeline activity to a blob storage or a sql table in ADF

Sampath S 81

I am using a scheduled trigger.
I want to save the output log from every execute pipeline activities to a blob storage or a sql table.
How can I do that?
The output should be saved to blob or table when ever the pipeline is success or failure with error message and details.
Can some one guide me?

@Suba Balaji

Sampath S 81 Reputation points

2022-10-02T08:00:04.127+00:00

@Suba Balaji Are you aware about this?
Suba Balaji 11,206 Reputation points

2022-10-02T08:10:36.13+00:00

Hi,

Can you elaborate requirement please, what's the expected output in case of success and failure?
Sampath S 81 Reputation points

2022-10-02T08:25:26.127+00:00

@Suba Balaji

On success
Pipelinename,status(success/failure),runid,pipeline run time
On failure
Pipelinename,status(success/failure),ErrorMessage,run id,pipelineruntime

I can rememeber this much parameters only.is it possible
Suba Balaji 11,206 Reputation points

2022-10-02T08:33:35.757+00:00

Yes it's possible.
Suba Balaji 11,206 Reputation points

2022-10-02T09:52:49.247+00:00

hi,

Same process holds. good for execute pipeline activity as well.

Please see below screenshots, i have an execute pipeline activity which calls pipeline3 where i am using a.fail activity to purposefully make it failed. If it fails i am assigning the error message to string variable. In your case you will either use Stored proc activity or copy activity to write to a table or blob. If succeeds, you will connect success flow (green box) to a set variable to set the message you want to write to a blob followed by copy activity.

Hope it helps.

Accepted answer

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-06T14:42:41.72+00:00

Hi @Sampath S ,

Thank you for posting query in Microsoft Q&A Platform. Let me share detailed steps here so that helps you to understand implementation details better.

In below Image, MainPipeline is the pipeline which actually runs my child pipeline using Execute Pipeline activity. I named my Execute Pipeline activity as ChildPipeline. Hence In all expressions You see ChildPipline as activity name. In your case when writing expressions kindly use your execute pipeline activity name accordingly.

Please check below video that explains how stored procedure activity will work also how to get error details and other details from an activity using expressions for better understanding.
Log Pipeline Executions to SQL Table using Azure Data Factory
Get Error message of Failed activities in Pipeline in Azure Data Factory

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer and Up-Vote buttons. Accepted answers help community as well.
Please sign in to rate this answer.

4 people found this answer helpful.
Sampath S 81 Reputation points

2022-10-06T15:05:45.627+00:00

@ShaikMaheer-MSFT
Thanks a lot This is what iam checking for

Sampath S 81 Reputation points

2022-10-06T18:01:59.993+00:00

@ShaikMaheer-MSFT I have one more query Can i get the child pipeline name inside (for example I am using a schedule trigger ,and in my pipline i have 30 execute pipeline activities ,after every execute pipline i need to update stored procedure like above which you mentioned ).Can i use the storred procedure only once i the pipiline so that i can get the job logs

Example.If i use my first execute pipeline activity for updating first sp,then how can i use it for 2nd executepipline.The parameters will change right?

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-07T04:18:53.833+00:00

Hi @Sampath S ,

Yes we have to use Stored Procedure activity for every Execute Pipeline activity. As you are rightly pointed parameters will change hence you cannot have only one SP activity for all Execute pipeline activities.

Hope this helps. Please consider hitting Accept Answer button and Up-Vote. Accepted answers help community as well. Also, Could you please consider retaking survey? Thank you.

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-07T12:14:33.22+00:00

Hi @Sampath S ,

Just checking if above answer helps. If yes, Please consider hitting Accept Answer button and consider to re-take survey. Accepted answers help community as well. Thank you.
Sign in to comment

1 additional answer

Suba Balaji 11,206 Reputation points

2022-10-02T08:44:35.863+00:00

Please check this blog url. It has the detailed steps with table design.

Please let us know if it helps.

Also a request to you, to add additional details of question in comment section instead of answer section as it might affect the question reach
Please sign in to rate this answer.

1 person found this answer helpful.
Sampath S 81 Reputation points

2022-10-02T08:50:56.057+00:00

@Suba Balaji Thank you Sure I will update.
But here I want to log the output after the execute pipeline activity,Not inside the pipeline.Is there any way for that.

Sampath S 81 Reputation points

2022-10-02T11:25:57.753+00:00

@Suba Balaji Is it possible.Please update once avaialble

Suba Balaji 11,206 Reputation points

2022-10-02T11:35:11+00:00

It's possible. I did answer but it's not being shown. let me check for few mins and again post the answer

Sampath S 81 Reputation points

2022-10-02T11:50:11+00:00

sure Please do Thank you for the help

Suba Balaji 11,206 Reputation points

2022-10-02T12:04:53.253+00:00

Hi,

The process is same for logging with pipeline or logging the execution using execute pipeline activity.

I tried with a sample pipeline.

My flow:

Pipeline 2 calls Pipeline 3 using Execute Activity.

Pipeline 3: I have a fail activity that purposefully fails the pipeline.

Error message shown in pipeline 3 is captured in pipeline 2 and assigned to a string variable.

After this step, you can either use copy activity (to log to sql.server or to blob) or use a stored proc activity to log to table.

Screenshots in the same order as i told above

Hope it helps.

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-06T09:00:56.04+00:00

Hi @Sampath S ,

Just checking in to see if the above answer helped. If this answers your query, do click and upvote for the same. And, if you have any further query do let us know.

Sampath S 81 Reputation points

2022-10-06T10:33:07.93+00:00

@Suba Balaji Actually in my case there is not like the op I want.When ever any activity got failed in child pipline I need to update the error in database.How can i do that?
Sign in to comment

Share via

Saving the output from a execute Pipeline activity to a blob storage or a sql table in ADF

1 additional answer