How to add a new field with a different record type using Avro in the schema registry?
Hi all,
Can another record be used as the type field in the Avro schema?
Example:
Existed schema:
{
"type": "record",
"name": "Foo",
"namespace": "com.example",
"fields": [
{
"name": "foo_string",
"type": "string"
}
]
}
Trying to create:
{
"type": "record",
"name": "Bar",
"namespace": "com.example",
"fields": [
{
"name": "foo",
"type": "com.example.Foo"
}
]
}
Currently, there is an error:
Failed to add schema 'com.example.Bar' with error: {"error":{"code":"InvalidRequest","message":"Avro schema validation failed: Undefined name: com.example.Foo at 'fields[0].type'. TrackingId:ebe0a8ba-d77d-4888-b50a-76d7ca2be3bc_G0, ...}}
Thanks in advance!
Azure Event Hubs
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-17T11:03:02.68+00:00 @Mariia Kharkhuta - Thanks for the question and using MS Q&A platform.
Yes, you can use another record as the type field in the Avro schema. However, you need to make sure that the referenced record is already registered in the schema registry before you can use it as a type in another record.
In your example, you are trying to reference the
com.example.Foo
record as the type of thefoo
field in thecom.example.Bar
record. However, it seems that thecom.example.Foo
record is not yet registered in the schema registry, which is why you are getting the error message.To fix this issue, you need to register the
com.example.Foo
record in the schema registry before you can use it as a type in thecom.example.Bar
record. Here's an example of how you can register thecom.example.Foo
record using the Azure CLI:az eventhubs schema registry create --resource-group <resource-group-name> --namespace-name <namespace-name> --schema-group <schema-group-name> --schema-name com.example.Foo --content '{"type":"record","name":"Foo","namespace":"com.example","fields":[{"name":"foo_string","type":"string"}]}'
Once the
com.example.Foo
record is registered in the schema registry, you can use it as a type in thecom.example.Bar
record without any issues.For more details, refer to Quickstart: Create an Azure Event Hubs schema registry using Azure portal and az eventhubs namespace schema-registry create.
Hope this helps. Do let us know if you any further queries.
-
Mariia Kharkhuta 0 Reputation points
2024-06-17T11:41:36.9833333+00:00 Thanks for the detailed answer! Unfortunately, no success has been achieved yet. The Foo scheme is registered in the Registry (via the UI), but when trying to register the Bar scheme, the same error occurs.
However, I tried to add the schema in the way you suggested, but I got the next informing message:
mariia ~ $ az eventhubs schema registry create --resource-group my-group --namespace-name my-namespace --schema-group my-schema-group --schema-name com.example.Foo --content '{"type":"record","name":"Foo","namespace":"com.example","fields":[{"name":"foo_string","type":"string"}]}' 'schema' is misspelled or not recognized by the system. Examples from AI knowledge base: https://aka.ms/cli_ref Read more about the command in reference docs
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-17T12:02:19.02+00:00 @Mariia Kharkhuta - Here's an example of how you can register the
com.example.Foo
record using the Azure CLI:az eventhubs schema-registry schema create --resource-group <resource-group-name> --namespace-name <namespace-name> --schema-group <schema-group-name> --schema-name com.example.Foo --content '{"type":"record","name":"Foo","namespace":"com.example","fields":[{"name":"foo_string","type":"string"}]}'
Once the
com.example.Foo
record is registered in the schema registry, you can use it as a type in thecom.example.Bar
record without any issues. If you are still getting the same error message, please make sure that thecom.example.Foo
record is registered in the correct schema group and namespace. -
Mariia Kharkhuta 0 Reputation points
2024-06-17T14:05:45.49+00:00 Should I install some specific extensions for the client?
mariia ~ $ az eventhubs schema-registry schema create --resource-group my-group --namespace-name my-namespace --schema-group my-schema-group --schema-name com.example.Foo --content '{"type":"record","name":"Foo","namespace":"com.example","fields":[{"name":"foo_string","type":"string"}]}' 'schema-registry' is misspelled or not recognized by the system. Examples from AI knowledge base: https://aka.ms/cli_ref Read more about the command in reference docs
Also, regarding schema group and namespace. I tried different namespaces:
e.g.
Schema Group: testSchemaGroupNamespace: testNamespace
1.1. Add Foo schema (Success){ "type": "record", "name": "Foo", "namespace": "com.example", "fields": [ { "name": "foo_string", "type": "string" } ] }
1.2. Add Bar schema (Fail)
{ "type": "record", "name": "Bar", "namespace": "com.example", "fields": [ { "name": "foo", "type": "com.example.Foo" } ] }
And
2.1. Add another Foo (after removing the previous one) (Success){ "type": "record", "name": "Foo", "namespace": "testSchemaGroup.testNamespace", "fields": [ { "name": "foo_string", "type": "string" } ] }
2.2. Add Bar schema (Fail)
{ "type": "record", "name": "Bar", "namespace": "testSchemaGroup.testNamespace", "fields": [ { "name": "foo", "type": "testSchemaGroup.testNamespace.Foo" } ] }
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-18T04:21:30.1566667+00:00 @Mariia Kharkhuta - You don't need to install any specific extensions for the Azure CLI to use the
az eventhubs schema registry
commands. However, you need to make sure that you have the latest version of the Azure CLI installed on your machine.Regarding the error you encountered, it seems that you misspelled the command. The correct command to create a new schema in the schema registry using the Azure CLI is:
az eventhubs schema registry create --resource-group <resource-group-name> --namespace-name <namespace-name> --schema-group-name <schema-group-name> --schema-name <schema-name> --content <schema-content>
Make sure to replace the placeholders with the actual values for your Azure Event Hubs instance.
Regarding the schema group and namespace, you can use any valid string as the schema group and namespace names. However, you need to make sure that the schema group and namespace names are unique within your Azure Event Hubs instance.
In your example, it seems that you created a new schema group and namespace for each schema. You can use the same schema group and namespace for all your schemas if they belong to the same logical group. This can make it easier to manage and organize your schemas.
Regarding the error you encountered when adding the "Bar" schema, it seems that you need to use the fully qualified name of the "Foo" schema in the "type" field of the "foo" field in the "Bar" schema. In your example, you used "testSchemaGroup.testNamespace.Foo" as the type, which is not a valid Avro type. Instead, you should use "com.example.Foo" as the type, which is the fully qualified name of the "Foo" schema.
Here's an example of how to create the "Bar" schema with the correct type:
{ "type": "record", "name": "Bar", "namespace": "testSchemaGroup.testNamespace", "fields": [ { "name": "foo", "type": "com.example.Foo" } ] }
Make sure to replace the <resource-group-name>, <namespace-name>, <schema-group-name>, and <schema-name> placeholders with the actual values for your Azure Event Hubs instance. If you encounter any issues, please let me know.
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-20T07:59:07.2366667+00:00 @Mariia Kharkhuta - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
Mariia Kharkhuta 0 Reputation points
2024-06-20T10:15:32.9966667+00:00 Hello! Sorry for the long answer. Unfortunately, none of the suggested results worked.Request:
az eventhubs schema registry create --resource-group <resource-group-name> --namespace-name <namespace-name> --schema-group-name <schema-group-name> --schema-name <schema- name> --content <schema-content> ```(latest client version: azure-cli version: 2.61.0) returns an error: ```sql 'schema' is misspelled or not recognized by the system.
It seems that the functionality of creating a scheme with an inline record is not implemented. If not, please provide a link to the documentation/example of how to use it.
What was tried:
- Azure CLI
- Azure Portal UI
The result is
"Avro schema validation failed: Undefined name: com.example.Foo at 'fields[0].type'.
- Schema registry API
Request:
curl --location --request PUT 'https://mynamespace.servicebus.windows.net/$schemagroups/mygroup/schemas/Bar?api-version=2023-07-01' \ --header 'Content-Type: application/json;serialization=Avro' \ --header 'Authorization: Bearer [TOKEN]' \ --data '{ "type": "record", "name": "Bar", "namespace": "com.example", "fields": [ { "name": "foo", "type": "com.example.Foo" } ] }'
Response:
{ "error": { "code": "InvalidRequest", "message": "Avro schema validation failed: Undefined name: com.example.Foo at 'fields[0].type'. TrackingId:980840fa-7c44-4f9a-891f-d860f5a7e3dc_G24, SystemTracker:mynamespace.servicebus.windows.net:$schemagroups/mygroup/schemas/Bar, Timestamp:2024-06-20T10:02:38" } }
The solution we will continue with is the embedded structure:
{ "type": "record", "name": "Bar", "namespace": "com.example", "fields": [ { "name": "foo", "type": { "type": "record", "name": "Foo", "fields": [ { "name": "foo_string", "type": "string" } ] } } ] }
But it's not scalable and does not fully fit our needs. If you have a better solution, please share.
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-24T07:39:37.15+00:00 @Mariia Kharkhuta - The command to create a schema with an inline record using the Azure CLI is:
az eventhubs schema-registry schema create --resource-group <resource-group-name> --namespace-name <namespace-name> --schema-group-name <schema-group-name> --schema-name <schema-name> --content '{"type": "record", "name": "Bar", "namespace": "com.example", "fields": [{"name": "foo", "type": {"type": "record", "name": "Foo", "namespace": "com.example", "fields": [{"name": "foo_string", "type": "string"}]}}]}'
This command should create a schema named
Bar
with an inline record namedFoo
in the specified schema group.If you still encounter issues with this command, you can try using the Azure Portal UI or the Schema Registry API to create the schema with an inline record.
Regarding the scalability of the embedded structure, it may not be the best solution if you have a large number of records with complex structures. In that case, you may want to consider defining the records in separate files and referencing them in your schema using the
{"type": "record", "name": "record-name", "namespace": "namespace", "aliases": ["alias"]}
syntax. This allows you to reuse the same record definition across multiple schemas. -
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-06-30T14:52:44.3+00:00 @Mariia Kharkhuta - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
Mariia Kharkhuta 0 Reputation points
2024-07-01T06:58:55.03+00:00 Hello! I really appreciate your help and time, but unfortunately, none of the solutions you suggested are working. The only possible option at the moment is the embedded schema (which does not meet our requirements).
-
PRADEEPCHEEKATLA-MSFT 89,816 Reputation points • Microsoft Employee
2024-07-02T04:40:21.52+00:00 @Mariia Kharkhuta - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Sign in to comment