Help to Return Only Non-Empty Array from Filter Activity in Set Variable Expression

Question

Hello,

I have a pipeline that retrieves metadata from a data lake, filters for files ending in "A", and then sets an external table creation script by concatenating the DDL, folder path, and file name with the suffix "A".

I'm encountering an error stating "...cannot be evaluated because array index '0' cannot be selected from an empty array." It appears that folders without an "A" are being returned as empty, which causes the error. Is there a method to return only the non-empty array element.

@concat(item().DDL,'''',variables('FolderPath'),activity('Filter for File Ending with A').output.value[0].name)

Answer 1

Here's how I solved this. To ensure I didn't return an empty filter activity array, the workaround is to use nested or functions. Below is the modified filter expression:

{

"type": "Expression",

"value": "@or(contains(item().name, 'M'), or(contains(item().name, '31'), contains(item().name, '30')))"

}

This ensures I get at least three files returned in the filter array output. Although, I still have the issue of picking the right file index in my set variable expression, but the above gets me closer to an ideal solution.

Answer 2

Hi @libpekin

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

I have a pipeline that retrieves metadata from a data lake, filters for files ending in "A", and then sets an external table creation script by concatenating the DDL, folder path, and file name with the suffix "A".

I'm encountering an error stating "...cannot be evaluated because array index '0' cannot be selected from an empty array." It appears that folders without an "A" are being returned as empty, which causes the error. Is there a method to return only the non-empty array element.

@concat(item().DDL,'''',variables('FolderPath'),activity('Filter for File Ending with A').output.value[0].name)

Solution:

Here's how I solved this. To ensure I didn't return an empty filter activity array, the workaround is to use nested or functions. Below is the modified filter expression:

{

"type": "Expression",

"value": "@or(contains(item().name, 'M'), or(contains(item().name, '31'), contains(item().name, '30')))"

}

This ensures I get at least three files returned in the filter array output. Although, I still have the issue of picking the right file index in my set variable expression, but the above gets me closer to an ideal solution.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.