Az ADF Periodic File exsistance check in Blob Storage

NR 1 Reputation point
2022-11-18T02:16:47.06+00:00

Hi Team,

I need to check for If blob storage container has 10 files and in case if container does not have 10 files then it has check for files after 15 minutes again in scheduled window. Ideally it should run for every 15 minutes in 2 hour window and if total 10 files are not available after 2 hour window then pipe line should fail.

Can you please assist me on how to do it.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,202 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Nandan Hegde 22,721 Reputation points
    2022-11-18T03:52:45.53+00:00

    Hey,
    create 2 variables>> counter and temp counter with values 0
    You can use an Untill activity
    within the untill activity :
    use getmeta data activity to get the number of files
    then use an if activity>> if number of files is 10 then nothing else use wait activity for 15 mins and then use set variable activity to add 1 to temp counter and then use set variable activity to assign counter variable the temp counter variable.

    untill activity condition would be either the number of files is 10 or counter variable value is 8 ( reason for 8 is 2 hr window with 15 mins intervals)
    after existing if the count is not 10 then use fail activity to throw error

    0 comments No comments

  2. Nandan Hegde 22,721 Reputation points
    2022-11-18T06:44:52.927+00:00

    261767-image.png

    untill logic:
    @or(equals(variables('Counter'),'4'),equals(string(length(activity('Get Metadata1').output.childItems)),'4'))

    untill activity:
    261771-image.png

    If condition:
    @equals(string(length(activity('Get Metadata1').output.childItems)),'4' )

    261658-image.png

    Tempcounter :
    @ hide (add(int(variables('Counter')),1))

    Counter:
    @variables('TempCounter')

    0 comments No comments