ADF unit test

Yang Chowmun 411

Is there any python version reference for azure data factory unit test?
So far I only found one uisng .NET https://richardswinbank.net/adf/set_up_automated_testing_for_azure_data_factory

Any suggestion will be greatly appreciate.

Accepted answer

MartinJaffer-MSFT 26,011 Reputation points

2021-09-13T17:55:39.167+00:00

Hello @Yang Chowmun and welcome to Microsoft Q&A.

After reading the blog entry you linked, I can summarize it as:

Using an API to create a pipeline run, and then check whether the pipeline run results return succeeded or failed.

This can be done in any of the SDK languages. Python is one of them. This is effectively programmatic monitoring on a test environment.

There is a quickstart which includes the necessary python calls.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

1 additional answer

PingpongSet 36 Reputation points

2022-02-21T14:32:49.657+00:00

Hi @MartinJaffer-MSFT

Based on what you said in your previous message quoted below, it is not unit tests, but belongs to other tests. Please clarify on this.

Using an API to create a pipeline run, and then check whether the pipeline run results return succeeded or failed.
Please sign in to rate this answer.
MartinJaffer-MSFT 26,011 Reputation points

2022-02-22T23:42:38.63+00:00

I agree with you, @PingpongSet , this is not unit test. In the context of Azure Data Factory, unit tests are not strictly possible.

Suppose we have a pipeline with 3 activities in series. In order, let us call them A, B, and C. (A -> B -> C)
We cannot run the activities in isolation. There is no functionality for that. This holds especially true when one relies on the output of another. Such a dependency probably breaks the definition of "unit test" anyway.
The Data Factory debug has a feature where we can stop execution after a chosen activity. This would let us run A, A&B, or A&B&C. We cannot run just B or just C or B&C. Pipeline execution starts at the beginning (A).

I suppose we could put each activity in separate pipelines, with mockups to handle inputs. This would be closer to the ideal of unit testing. We can retrieve Activity Run details given a Pipeline Run ID. This provides data similar to debug runs, but in a programatically available format.

In general, when ti comes to Data Factory, a different sort of testing is used.

I see you made a related question post @PingpongSet . Do you still desire attention there?
Sign in to comment