How to Use Wildcards in Data Flow Source Activity?

Question

In Data Factory I am trying to set up a Data Flow to read Azure AD Signin logs exported as Json to Azure Blob Storage to store properties in a DB. The problem arises when I try to configure the Source side of things. No matter what I try to set as wild card, I keep getting a "Path does not resolve to any file(s). Please make sure the file/folder exists and is not hidden.".

I use the Dataset as Dataset and not Inline. I can click "Test connection" and that works. In the Source Tab and on the Data Flow screen I see that the columns (15) are correctly read from the source and even that the properties are mapped correctly, including the complex types.

The actual Json files are nested 6 levels deep in the blob store. Is that an issue?

I've tried all kinds of wildcards:

• **
• /**/*.Json
• //////*.json
• tenantId*/*.json
• *.json

None of it works, also when putting the paths around single quotes or when using the toString function.

Does anyone know if this can work at all?

Answer 1

@MartinJaffer-MSFT - thanks for looking into this. It proved I was on the right track. I've now managed to get json data using Blob storage as DataSet and with the wild card path you also have. The underlying issues were actually wholly different:

• Needed to disable soft delete: https://learn.microsoft.com/en-us/answers/questions/472879/azure-data-factory-data-flow-with-managed-identity.html
• Automatic schema inference did not work; uploading a manual schema did the trick
• Eventually I moved to using a managed identity and that needed the Storage Blob Reader role. Account Keys and SAS tokens did not work for me as I did not have the right permissions in our company's AD to change permissions.

It would be great if the error messages would be a bit more descriptive, but it does work in the end.

Answer 2

@Raimond Kempees

Given a filepath
tenantId=XYZ/y=2021/m=09/d=03/h=13/m=00/anon.json

I was able to see data when using inline dataset, and wildcard path

*/*/*/*/*/*/*.json