Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,726 questions
ADF - Filter Activity output to Copy Activity
Anonymous
We have a requirement to filter the API response and load the filter data into different cosmos containers. used webActivity, Filter Activity and CopyActivity. as below. ![232277-image.png][1] Copy Activity source configured as below. ![232317-image.png][2] The out come of filter activity is below. need to load all objects inside the value array { "source": { "type": "CosmosDbSqlApiSource", "additionalColumns": [ { "name": "JsonOutput", "value": [ { "ApplicationAdmitYear": 2021, "Country": "US", "ApplicationDate": "2021-05-12T00:00:00Z ", "ApplicationNumber": 12345, "ApplicationProgressStatus": [ { "ActionDate": { "DateTime": "2021-05-13T00:00:00Z " } } ] }, { "ApplicationAdmitYear": 2021, "Country": "AUS", "ApplicationDate": "2021-05-12T00:00:00Z ", "ApplicationNumber": 12345, "ApplicationProgressStatus": [ { "ActionDate": { "DateTime": "2021-05-13T00:00:00Z " } } ] } ] } ] } } Getting below error in Copy activity , any suggestions and help please Details ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'value' property to 'System.String' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.InvalidCastException,Message=Object must implement IConvertible.,Source=mscorlib,' [1]: /api/attachments/232277-image.png?platform=QnA [2]: /api/attachments/232317-image.png?platform=QnA
{count} votes
-
Vinod Kumar 246 Reputation points
2022-08-18T13:37:52.177+00:00 Hi anonymous user,
If you want to store data into cosmos then it should be your sink. And also I believe the response you have shared is the output of web activity.
Thanks @Vinod Kumar looking into it. Below the source API response { "metadata": [ { "count": 1000, "page_num": 1, "page_count": 6, "total_count": 5225 } ], "data": [ { "ApplicationAdmitYear": 2021, "Country": "US", "ApplicationDate": "2021-05-12T00:00:00Z ", "ApplicationNumber": 12345, "ApplicationProgressStatus": [ { "ActionDate": { "DateTime": "2021-05-13T00:00:00Z " } } ] }, { "ApplicationAdmitYear": 2021, "Country": "UF", "ApplicationDate": "2021-05-12T00:00:00Z ", "ApplicationNumber": 4567, "ApplicationProgressStatus": [ { "ActionDate": { "DateTime": "2021-05-13T00:00:00Z " } } ] } ]} and this one is filter activity output { "ItemsCount": 1000, "FilteredItemsCount": 247, "Value": [ { "ApplicationAdmitYear": 2021, "Country": "US", "ApplicationDate": "2021-05-12T00:00:00Z ", "ApplicationNumber": 12345, "ApplicationProgressStatus": [ { "ActionDate": { "DateTime": "2021-05-13T00:00:00Z " } } ] }]} what i was trying to do is , the "Value" array of the filter activity output trying to load in cosmos. For that i used copy activity. In copy activity source , i didnt find the option to use filter activity output hence taken dummy source (not return any data) with additional column and the logic is @activity('Filter US Data').output.value . ![232486-image.png][1] ![232460-image.png][2] [1]: /api/attachments/232486-image.png?platform=QnA [2]: /api/attachments/232460-image.png?platform=QnA
Thanks, @Vinod Kumar
Hi anonymous user,
Welcome to the MS Q&A platform.
Were you able to find out the issue, or please let us know if you need any further help here?
Hi @BhargavaGunnam-MSFT
The issue is not fixed . Struggling to pass the filter activity output value to copy activity as a input.
Any suggestions?
Hi anonymous user,
As per the error message, It seems like the data conversion issue. Could you please try changing the data type at the cosmos end and re-import the schema and mapping in the copy activity?
Hi anonymous user,
I am just checking in to see if you have any updates here.
Sign in to comment