Azure Event Hubs with Azure Function - Serialization failed due to an unsupported type, System.Byte[]

Eric H 5

I am getting this error sometimes my function gets an event from azure event hubs:
Azure.Core.Amqp: Serialization failed due to an unsupported type, System.Byte[]

I can go to azure and generate an event, and it works fine, the EventData[] gets bound properly, and I can step through my function.

But any events created by our Kafka Change Data Capture end up triggering this exception (on this single consumer).

In the erroring consumer, I can change the "EventData[]" to "string[]" and the function runs, but then I lose the "partitionkey" information that I want.

We have other azure function consumers consuming the same event, with what appears to be the same basic code, and the bindings all work fine using EventData[].

Function signature (this one works):
public static async Task RunMaster([EventHubTrigger("%MasterEventHubName%", Connection = "EventHubConnectionMaster")] EventData[] eventsNames, ILogger log)

Here is the function signature that doesn't work:

public async Task Run([EventHubTrigger("%EventHub:Subscribe:Name%", Connection = "EventHub:Subscribe:SecretAuth:ConnectionString")] EventData[] events, ILogger log)

Both projects are using Azure.Messaging.EventHubs >6.0

The payload of the event only has simple properties (int and string).

What could be causing that serialization error?

Thank you,

Eric

PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee

2024-07-16T07:50:45.5133333+00:00
@Eric H - Thanks for the question and using MS Q&A platform.

It seems like the issue is with the serialization of the payload of the event. The error message indicates that the type System.Byte[] is not supported.

One possible reason for this error is that the payload of the event contains a property of type System.Byte[]. You can try to check the payload of the events created by your Kafka Change Data Capture to see if this is the case.

Another possible reason is that the payload of the event is not compatible with the EventData type. In this case, you can try to change the type of the events parameter to string[] and use a custom serializer to deserialize the payload. You can then extract the partitionkey information from the payload and pass it to the EventData constructor.

Here is an example of how you can use a custom serializer to deserialize the payload:

public async Task Run([EventHubTrigger("%EventHub:Subscribe:Name%", Connection = "EventHub:Subscribe:SecretAuth:ConnectionString")] string[] events, ILogger log) { foreach (var eventData in events) { var payload = JsonConvert.DeserializeObject<MyPayload>(eventData); var eventBody = Encoding.UTF8.GetBytes(payload.Body); var eventProperties = new Dictionary<string, object> { { "partitionkey", payload.PartitionKey } }; var eventData = new EventData(eventBody) { Properties = eventProperties }; // process the eventData } } public class MyPayload { public string Body { get; set; } public string PartitionKey { get; set; } }

In this example, the payload of the event is deserialized into a custom class MyPayload, which has properties for the Body and PartitionKey. The Body property is then used to create the eventBody byte array, and the PartitionKey property is added to the eventProperties dictionary. Finally, the EventData constructor is used to create the eventData object, which can be processed by the function.

Hope this helps. Do let us know if you any further queries.
Eric H 5 Reputation points

2024-07-16T16:10:50.2333333+00:00

Hello, I appreciate the response, but I don't think it's a data issue.

The payload of the event has nothing but integer and string properties.

This same event IS able to be bound to "EventData" with different functions.

From what I can tell, the functions that ARE working are using "dotnet" while the function that doesn't work is using "dotnet-isolated".

So it appears to be something about the "isolated". I followed the guidance for isolated functions https://learn.microsoft.com/en-us/azure/azure-functions/dotnet-isolated-process-guide?tabs=windows and it says that for EventHubs in an isolated process, I can use "EventData[]" https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs?tabs=isolated-process%2Cextensionv5&pivots=programming-language-csharp#binding-types
I think I've followed all of the guidance. A non-isolated function can consume the event just fine, but an isolated function gets a binding error on "EventData[]".

I created a brand-new function from scratch using .net 8, and it is able to bind "EventData[]" just fine. I created a brand-new function from scratch using .net 8 isolated, and it also is unable to bind the event to "EventData[]", I get the same error.

How can I get that binding to be valid using dotnet-isolated?
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee

2024-07-17T05:58:34.53+00:00

@Eric H - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Tom Annetts 0 Reputation points

2024-07-18T08:18:43.9533333+00:00

Having the exact same issue migrating to isolated process.

Seems others are having the same issue:

https://github.com/Azure/azure-functions-dotnet-worker/issues/2481

Raised almost a couple of months ago, not sure anything has been done with it yet but seems a bit of a major bug?
Mitalben Shah (US) 0 Reputation points

2024-07-30T19:46:51.7+00:00

I am also having similar issue, and I have migrated from in-process dotnet 6 to isolated dotnet 8. Does anyone have any workaround for this?

Share via

Azure Event Hubs with Azure Function - Serialization failed due to an unsupported type, System.Byte[]

Your answer