AVRO connecctor in ADF doesnt work

Question

AVRO connecctor in ADF doesnt work

Hi Expers, I have an event hub, but since ADF connector to event hub is not available, I tried the option is event hub to write data to an azure blob store. But it supports only writing this event data in AVRO format in Blob store. I tried using the native ADF connector for this AVRO file, but it is not able to read the file. I tried all the compression available, snappy deflate etc but nothing seems to work. Below is the error message I am getting. Column: SystemProperties,Location: Source,Format: Avro,The data type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' is currently not supported by Avro format. . Activity ID: dd332d01-f7c2-4737-96af-ff0801242569 Attached is the sample file I tried for testing. Avro format is not supported in the forum attachement, so saving it as txt and attaching it here.[50711-eventhubsample.txt][1] [1]: /api/attachments/50711-eventhubsample.txt?platform=QnA Thanks, Nikesh

MarkKromer-MSFT 5,146 Reputation points Microsoft Employee

2020-12-23T20:16:28.27+00:00

You should be able to read it as a Source in Data Flows since Data Flow uses Spark and can read complex data types. For the dataset, make sure that you clear out the schema in the dataset first. https://techcommunity.microsoft.com/t5/azure-data-factory/read-and-write-complex-data-types-in-adf/ba-p/1772763
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-24T06:26:08.16+00:00

Hi anonymous user,

I am unable to get the file you shared in the data flow to read as well but it gives me a different error. And, I am able to do it fine with sample avro files I have downloaded from GitHub. To assist you better, can you please share the actual .avro file to AzCommunity[at]microsoft[dot]com with subject - "Attn:Haritha - AVRO connecctor in ADF doesnt work".

Looking forward for your response!
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-24T10:46:55.693+00:00

Hi anonymous user,

Thanks for reaching out. As discussed in the mail, the issue seems to be happening only with preview of data but the "import projection" and pipeline when ran are successful. I will reach out to product team for the preview feature for AVRO files and will get back to you with insights.
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-28T12:32:42.37+00:00

Hi anonymous user,

Due to a holiday season, there is a delay in getting the response from the product team. I will get back to you as soon as I hear from them.

Thanks for your patience!

MarkKromer-MSFT 5,146 Reputation points Microsoft Employee

2020-12-23T20:16:28.27+00:00

You should be able to read it as a Source in Data Flows since Data Flow uses Spark and can read complex data types. For the dataset, make sure that you clear out the schema in the dataset first. https://techcommunity.microsoft.com/t5/azure-data-factory/read-and-write-complex-data-types-in-adf/ba-p/1772763
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-24T06:26:08.16+00:00

Hi anonymous user,

I am unable to get the file you shared in the data flow to read as well but it gives me a different error. And, I am able to do it fine with sample avro files I have downloaded from GitHub. To assist you better, can you please share the actual .avro file to AzCommunity[at]microsoft[dot]com with subject - "Attn:Haritha - AVRO connecctor in ADF doesnt work".

Looking forward for your response!
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-24T10:46:55.693+00:00

Hi anonymous user,

Thanks for reaching out. As discussed in the mail, the issue seems to be happening only with preview of data but the "import projection" and pipeline when ran are successful. I will reach out to product team for the preview feature for AVRO files and will get back to you with insights.
HarithaMaddi-MSFT 10,081 Reputation points

2020-12-28T12:32:42.37+00:00

Hi anonymous user,

Due to a holiday season, there is a delay in getting the response from the product team. I will get back to you as soon as I hear from them.

Thanks for your patience!

Answer 1

HarithaMaddi-MSFT 10,081

Hi anonymous user,

Thanks for your patience again! I got an update from the product team that currently Azure data factory do not fully support Dictionary or Map types in Avro and this feature gap is taken as a bug by the team. They are working on it and will be implemented in future releases.

Apologies for the inconvenience. Since the data can be loaded into sink, we hope it is not a blocker for implementation using dataflow. Please suggest if there is any specific challenge due to this limitation and we will provide the alternative solution for the same.

--

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

HarithaMaddi-MSFT 10,081 Reputation points

2020-12-30T12:32:04.937+00:00

Hi anonymous user,

We have not received a response from you. Please suggest if above suggested approach is helpful. Otherwise, let us know and we will continue to engage with you on the query.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members.
Vadavathi, Nikesh (Cognizant) 21 Reputation points

2020-12-30T16:13:16.107+00:00

Thank you, I am able to parse the AVRO file using maping data flow. Thanks for your timely support

AVRO connecctor in ADF doesnt work

0 additional answers

Activity