Validation activity - checking the existence of a single file

Kothai Ramanathan 941 Reputation points Microsoft Employee
2020-12-08T07:57:59.073+00:00

I have a validation activity. In this activity, I want to check if a particular blob file exists or not. I provide the container name, directory name and the file name as shown below.

46109-image.png

This timesout even if the file exists. However, if I delete the file name parameter, it succeeds.

Can the validation activity be used to check the existence of a single file, or can it be used only at a folder level ? Since the filename can be provided as a parameter, I assume that we can check the existence of a single file. Please let me know if I am missing something here.

Thanks,
Kothai.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jaryd 181 Reputation points
    2020-12-10T03:54:40.38+00:00

    SaurabhSharma-msft's answer looks great and I am going to try and walk through those steps.

    However, previously I had trouble getting the validation activity to work as I wanted.

    I wound up using a 'Get Metadata' activity. I add the 'Child Items' Argument. (I'm also using the 'Filter by last modified' to only look at files from the last day. Also, my dataset a Binary Dataset from an SFTP Linked Service)

    I then use an 'If Condition' activity to check if the file name was in the Get Metadata Child Item results.

    My 'If Condition' Expression is: 

    @greaterOrEquals(indexof(string(activity('<Get Metadata Activity Name>').output.childItems),'<file name>'),0)
    1 person found this answer helpful.
    0 comments
  2. Saurabh Sharma 23,751 Reputation points Microsoft Employee
    2020-12-10T02:11:52.907+00:00

    @Kothai Ramanathan Thanks for using Microsoft Q&A.

    Yes validation activity can be used to check the existence of a file as well. Sorry but I am unable to reproduce your issue and it works for me if filename provided is available at the container end. (Please find the gif below where it checks if the emp.txt file exists).
    46782-validationactivity.gif

    Container Screenshot
    46783-image.png

    You can find another gif below when the file doesn't exists on the storage. I have reduced the timeout period to 30 s for the validation activity to make the pipeline concludes without waiting long which also works as expected.
    46697-validationactivity-2.gif

    Can you please make sure that you are passing the correct filename to @dataset().FileName parameter. Passing incorrect filename may be causing your validation activity to wait till timeout happens.
    Please let me know if that's not the case.