Did something change with GetMetadata and Wild Cards in Azure Data Factory?

Question

Before last week a Get Metadata with a wildcard would return a list of files that matched the wildcard.

Now I'm getting the files and all the directories in the folder. The directory names are unrelated to the wildcard. I now have to filter the list! Is anyone else seeing this and is it intentional?

Answer 1

Hello @Wilson, James S and welcome to Microsoft Q&A. Thank you very much for your question.

Initially I was going to say, Get Metadata never worked that way, and that getting child items of a file, rather than a folder did not make sense. However, I gave a number of combinations a try, and here is what I found:

Given a file/blob based dataset, with the container defined, and the file parameterized (see below picture),

and a Get Metadata activity using this dataset, fetching childItems,

If the Start Time, or the End Time, or both are used, then the output does filter the items using the wildcard. Only files show up in the output.

If neither Start Time nor End Time are used, (and instead left blank,) then the output filters files by wildcard, but also includes all folders unfiltered.

Thank you for asking me to look into this, I learned something today. If you are certain it worked differently before (an example pipeline run ID would help), I will make further inquiries into this issue.

Answer 2

In an ironic twist of fate, this seems to have stopped working yesterday. @Wilson, James S @MartinJaffer-MSFT

Can either of you confirm?

Answer 3

So there was an unintended feature at one point in time.