Edit

Validation activity in Azure Data Factory and Synapse Analytics pipelines

  • Article

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

You can use a Validation in a pipeline to ensure the pipeline only continues execution once it has validated the attached dataset reference exists, that it meets the specified criteria, or timeout has been reached.

Create a Validation activity with UI

To use a Validation activity in a pipeline, complete the following steps:

  1. Search for Validation in the pipeline Activities pane, and drag a Validation activity to the pipeline canvas.
  2. Select the new Validation activity on the canvas if it is not already selected, and its Settings tab, to edit its details. Screenshot shows the UI for a Validation activity.
  3. Select a dataset, or define a new one by selecting the New button. For file based datasets like the delimited text example above, you can select either a specific file, or a folder. When a folder is selected, the Validation activity allows you to ignore validation of the existence of child items in the folder, or require whether child items exist or not.
  4. The output of the Validation activity can be used as an input to any other activities, and referenced within those activities for any of their properties using dynamic expressions.

Syntax


{
"name": "Validation_Activity",
"type": "Validation",
"typeProperties": {
"dataset": {
"referenceName": "Storage_File",
"type": "DatasetReference"
},
"timeout": "0.12:00:00",
"sleep": 10,
"minimumSize": 20
}
},
{
"name": "Validation_Activity_Folder",
"type": "Validation",
"typeProperties": {
"dataset": {
"referenceName": "Storage_Folder",
"type": "DatasetReference"
},
"timeout": "0.12:00:00",
"sleep": 10,
"childItems": true
}
}

Type properties

Property Description Allowed values Required
name Name of the 'Validation' activity String Yes
type Must be set to Validation. String Yes
dataset Activity will block execution until it has validated this dataset reference exists and that it meets the specified criteria, or timeout has been reached. Dataset provided should support "MinimumSize" or "ChildItems" property. Dataset reference Yes
timeout Specifies the timeout for the activity to run. If no value is specified, default value is 12 hours ("0.12:00:00"). Format is d.hh:mm:ss String No
sleep A delay in seconds between validation attempts. If no value is specified, default value is 10 seconds. Integer No
childItems Checks if the folder has child items. Can be set to-true : Validate that the folder exists and that it has items. Blocks until at least one item is present in the folder or timeout value is reached.-false: Validate that the folder exists and that it is empty. Blocks until folder is empty or until timeout value is reached. If no value is specified, activity will block until the folder exists or until timeout is reached. Boolean No
minimumSize Minimum size of a file in bytes. If no value is specified, default value is 0 bytes Integer No

Related content

See other supported control flow activities: