Filter activity Failing when input is empty in ADF

Question

Filter activity Failing when input is empty in ADF

Prapul Kumar Dongari 116

Hi team , In my pipeline I'm using filter activity to filter desired items from json when input to filter is output of another filter activity(which returned out with empty array) as attached in screen shot. please help me how to handle failure. User's image

output of first filter: User's image

error message at second filter: User's image

Harishga 6,005 Reputation points Microsoft External Staff

2024-02-02T11:37:03.3333333+00:00

Hi @Prapul Kumar Dongari
Welcome to Microsoft Q&A platform and thanks for posting your question here.

Based on the error message you provided, it seems that the issue is with an expression trying to evaluate parameters from an empty array. One way to handle this issue is to add a condition to check if the array is empty before trying to access its elements. You can use the If Condition activity to check if the array is empty and then use the Set Variable activity to set a variable to true or false based on the result of the condition. You can then use this variable to control the flow of your pipeline. If the array is not empty, you can proceed with the filter activity.

Another option is to use the Try-Catch block approach. In this approach, you define the business logic and only define the Upon Failure path to catch any error from the previous activity.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-fail-activity
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling

I hope this information helps you. Let me know if you have any further questions or concerns.
Prapul Kumar Dongari 116 Reputation points

2024-02-02T11:43:27.66+00:00

Hi @Harishga , Thanks for your reply but I don't want to use if activity in my case i want to handle that in filter condition query only. adding if condition and extra activities it will become complexity.
Harishga 6,005 Reputation points Microsoft External Staff

2024-02-02T12:59:11.46+00:00
@Prapul Kumar Dongari
In that case, you can modify the filter condition to handle the scenario where the input array is empty. You can use the empty function to check if the array is empty and then use the or operator to return a default value if the array is empty. Here's an example:

@or(empty(items('FirstFilterActivity')), <your filter condition>)

In this example, items('FirstFilterActivity') is the output of your first filter activity, and <your filter condition> is the condition you want to apply to the input array. The empty function checks if the input array is empty, and the or operator returns the default value if the input array is empty. You can replace <your filter condition> with your actual filter condition.
Reference
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions
I hope this information helps you. Let me know if you have any further questions or concerns.
Prapul Kumar Dongari 116 Reputation points

2024-02-05T03:04:15.2066667+00:00

Hi, I tried this I'm unable to use this expression can you please provide the screenshot of ADF activities which you tried.
Prapul Kumar Dongari 116 Reputation points

2024-02-05T03:29:08.84+00:00
Hi tried with new logic its working as expected.

@if( empty(activity('previous_act_name').output.value), json('[]'), activity('previous_act_name').output.value[0].Params )
Harishga 6,005 Reputation points Microsoft External Staff

2024-02-05T04:52:04.9366667+00:00

@Prapul Kumar Dongari Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Answer accepted by question author

0 additional answers

Your answer

Harishga 6,005 Reputation points Microsoft External Staff

2024-02-02T11:37:03.3333333+00:00

Hi @Prapul Kumar Dongari
Welcome to Microsoft Q&A platform and thanks for posting your question here.

Based on the error message you provided, it seems that the issue is with an expression trying to evaluate parameters from an empty array. One way to handle this issue is to add a condition to check if the array is empty before trying to access its elements. You can use the If Condition activity to check if the array is empty and then use the Set Variable activity to set a variable to true or false based on the result of the condition. You can then use this variable to control the flow of your pipeline. If the array is not empty, you can proceed with the filter activity.

Another option is to use the Try-Catch block approach. In this approach, you define the business logic and only define the Upon Failure path to catch any error from the previous activity.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-fail-activity
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-failure-error-handling

I hope this information helps you. Let me know if you have any further questions or concerns.
Prapul Kumar Dongari 116 Reputation points

2024-02-02T11:43:27.66+00:00

Hi @Harishga , Thanks for your reply but I don't want to use if activity in my case i want to handle that in filter condition query only. adding if condition and extra activities it will become complexity.
Harishga 6,005 Reputation points Microsoft External Staff

2024-02-02T12:59:11.46+00:00

@Prapul Kumar Dongari
In that case, you can modify the filter condition to handle the scenario where the input array is empty. You can use the empty function to check if the array is empty and then use the or operator to return a default value if the array is empty. Here's an example:

@or(empty(items('FirstFilterActivity')), <your filter condition>)

In this example, items('FirstFilterActivity') is the output of your first filter activity, and <your filter condition> is the condition you want to apply to the input array. The empty function checks if the input array is empty, and the or operator returns the default value if the input array is empty. You can replace <your filter condition> with your actual filter condition.
Reference
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions
I hope this information helps you. Let me know if you have any further questions or concerns.
Prapul Kumar Dongari 116 Reputation points

2024-02-05T03:04:15.2066667+00:00

Hi, I tried this I'm unable to use this expression can you please provide the screenshot of ADF activities which you tried.
Prapul Kumar Dongari 116 Reputation points

2024-02-05T03:29:08.84+00:00

Hi tried with new logic its working as expected.

@if( empty(activity('previous_act_name').output.value), json('[]'), activity('previous_act_name').output.value[0].Params )
Harishga 6,005 Reputation points Microsoft External Staff

2024-02-05T04:52:04.9366667+00:00

@Prapul Kumar Dongari Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Answer 1

Hi Prapul Kumar Dongari
Welcome to Microsoft Q&A platform and thanks for posting your question. I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer. Ask: In my pipeline I'm using filter activity to filter desired items from json when input to filter is output of another filter activity (which returned out with empty array) Solution:

Hi tried with new logic its working as expected.

SQL

AI Convert

Copy
@if(
    empty(activity('previous_act_name').output.value),
    json('[]'),
    activity('previous_act_name').output.value[0].Params
)

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Filter activity Failing when input is empty in ADF

0 additional answers

Your answer