Azure Event Hubs schema registry validation failure with union types when re-registering a schema

Question

In an Azure Schema Registry in Event Hubs, I get an error when re-registering a schema with the exact same schema definition when that schema definition contains a field that is a union of other records.

Here is an example using the Azure java SDK:

var schemaDefinition = """
    {
      "type": "record",
      "name": "TestRecord",
      "fields": [{
        "name": "union_field",
        "type": [{
          "type": "record",
          "name": "RecordA",
          "fields": [{
            "name": "foo",
            "type": "string"
          }]
        }, {
          "type": "record",
          "name": "RecordB",
          "fields": [{
            "name": "bar",
            "type": "string"
          }]
        }]
      }]
    }
    """;

var client = new SchemaRegistryClientBuilder()
    .fullyQualifiedNamespace(namespace)
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildClient();

// first call succeeds
client.registerSchema(schemaGroup, schemaName, schemaDefinition, SchemaFormat.AVRO);
// second call fails
client.registerSchema(schemaGroup, schemaName, schemaDefinition, SchemaFormat.AVRO);

The first call to registerSchema successfully registers the schema, and I am able to see it on the Azure Portal. The second call fails with a 400 status and returns the error message Json schema validation failed: An item with the same key has already been added. I see a similar message when trying to modify the schema via the Azure Portal web UI, making it impossible to publish new versions of the schema.

According to the API documentation for registerSchema, if there is a matching schema then "the id of that schema is returned"

Is there something incorrect in the schema definition? Or is there an issue with the API?

Answer 1

Hi Jing H,

Thank you for posting query in Microsoft Q&A Platform.

Based on the error message you received, it seems that the issue is related to the schema definition. The error message "Json schema validation failed: An item with the same key has already been added" indicates that there is a duplicate key in the schema definition.

In your example, the "union_field" is defined as a union of two records, "RecordA" and "RecordB". However, the "name" field in both records is the same, which is causing a conflict. To fix this issue, you can rename the "name" field in one of the records to a different name.

Here is an updated schema definition with the "name" field in "RecordB" renamed to "baz":

Java

Copy var schemaDefinition = """ { "type": "record", "name": "TestRecord", "fields": [{ "name": "union_field", "type": [{ "type": "record", "name": "RecordA", "fields": [{ "name": "foo", "type": "string" }] }, { "type": "record", "name": "RecordB", "fields": [{ "name": "baz", "type": "string" }] }] }] } """;

After making this change, you should be able to register the schema without any issues.

Please try above and let me know how it goes.