Is it possible to use a wild card inside a dataset?

Question

I am trying to use wildcard (in expression) inside a dataset. Is it possible?

Basically, here is process:

1. Use "Get Metadata" to look for several csv files inside certain folder (within Azure File Share).
2. If Structure matches with other one reference csv file (meaning if the file has a header/no header), it will either pass/no pass (using "If Condition").

But, I am stuck at getting the wildcard set up using Expression (dynamic content) inside a dataset.

The error message is this:

Please make sure the file path and filename does not contain special characters. The specified resource name contains invalid characters.

I was basically using something like this expression (some character deleted for confidentiality):

@concat('Check_Detail Center LLC','*','.csv')

If I use the Wildcard file path (using '*') at "Copy data", it works.

But, if I try to use wild card at dataset (inside "Get Metatdata"), it does not work.

Expression itself seems to be fine as it does not give an error inside "Pipeline expression builder".

Is there a way to use wildcard inside dataset?

Thanks.

Answer 1

Hey,

Can you please share an image of your entire ADF pipeline dataflow and specify what you are trying to achieve?

Because based on what I could understand from the current aspect, you are right that dataset doesnt accept wildcard aspects.

So what you can try is via get meta data activity get the list of all files within your path and then use filter activity to get the list of files that you need based on wildcard character.

Answer 2

Hi King Java ,

Thankyou for posting your query on Microsoft Q&A platform. It seems you are trying to use a wildcard in an expression inside a dataset in Azure Data Factory. Additionally, I understand that you want to compare the file structure and based on that you want to consider or skip the file for data load. Please correct me if my understanding is wrong.

Unfortunately, wild card pattern can't be used inside dataset. You can only parameterize the dataset to pass filenames dynamically.

Do you have definite column names or it would be dynamic?

If it's defined, then you can use 'structure' as the field list in get metadata to get the column names . In If block you can do the comparison, if structure array contains the column names you are looking for then go ahead with the data load.

Helpful resources:

Wildcard filters in copy activity of ADF

List of files in copy activity of adf

Kindly share insights on the above required details, in case you have further questions , or else, accept the answer by clicking on Accept answer button.