Azure Event Hubs
An Azure real-time data ingestion service.
719 questions
Difference between schema.history and database.history in mysql connector
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 93%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EET%3C/text%3E%3C/svg%3E)
Eddy Tran
40
Reputation points
I set up kafka, mysql and debezium in my computer successfully. However, i have still not understood about the difference between database.history.kafka.topic and schema.history.internal.kafka.topic. Below is my connector:
Please help me explaining the difference between database.history.kafka.topic and schema.history.internal.kafka.topic! Why when i run kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, the result is schemahistory.dbserver1; what about dbhistory.classicmodels, where can i find topic dbhistory.classicmodels?
Many thanks!
Azure Event Hubs
Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
397 questions
{count} votes
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator2024-01-17T03:32:10.2566667+00:00 @Eddy Tran - Thanks for the question and using MS Q&A platform.
Could you please help us understand which Azure service are you using?
-
Eddy Tran 40 Reputation points
2024-01-17T03:47:01.6066667+00:00 I'm using Azure Event Hub. Thanks!
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2024-01-17T04:48:51.08+00:00 @Eddy Tran - Thanks for the details.
The
database.history.kafka.topicandschema.history.kafka.topicare two different topics used by Debezium to store the history of changes in the database schema and data respectively.The
database.history.kafka.topicis used to store the history of changes in the database schema. This topic is used to store the DDL statements that are executed on the database. This topic is used to store the history of changes in the database schema. This topic is used to store the DDL statements that are executed on the database.The
schema.history.kafka.topicis used to store the history of changes in the database data. This topic is used to store the changes in the data that are captured by Debezium.When you run the command
kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, the result isschemahistory.dbserver1because this is the default name of the topic used by Debezium to store the history of changes in the database schema. Regarding thedbhistory.classicmodelstopic, it is not created by default. You can configure Debezium to use a different topic name for thedatabase.history.kafka.topicproperty. If you have configured Debezium to usedbhistory.classicmodelsas thedatabase.history.kafka.topic, you can find this topic by running the same commandkafka-topics.bat --list --bootstrap-server 127.0.0.1:9092.Hope this helps.
-
Eddy Tran 40 Reputation points
2024-01-18T03:24:23.7333333+00:00 Actually, when i configure the connector with schema.history.kafka.topic and database.history.kafka.topic successfully. When i run kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, I still not see the database.history.kafka.topic.
Thanks!
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2024-01-18T05:10:17.69+00:00 @Eddy Tran - If you have configured the
database.history.kafka.topicproperty in your connector configuration, but you are not seeing the topic when you run thekafka-topics.bat --list --bootstrap-server 127.0.0.1:9092command, there could be a few reasons for this:The topic has not been created yet: Debezium will automatically create the
database.history.kafka.topictopic when it starts up and detects that the topic does not exist. If you have not yet started the connector, or if the connector has not yet executed any DDL statements, the topic may not have been created yet. You can check if the topic has been created by running thekafka-topics.bat --describe --bootstrap-server 127.0.0.1:9092 --topic database.history.kafka.topiccommand.The topic has been created with a different name: If you have specified a custom name for the
database.history.kafka.topicproperty in your connector configuration, the topic may have been created with a different name. You can check the actual name of the topic by running thekafka-topics.bat --describe --bootstrap-server 127.0.0.1:9092 --topic <actual-topic-name>command, replacing<actual-topic-name>with the name of the topic that you specified in your configuration.There is an issue with the configuration: If there is an issue with the configuration of your connector, Debezium may not be able to create the
database.history.kafka.topictopic. You can check the logs of the connector to see if there are any errors related to the topic creation.I hope this helps! Let me know if you have any further questions.
-
PRADEEPCHEEKATLA 90,641 Reputation points Moderator
2024-01-25T04:42:15.9933333+00:00 @Eddy Tran - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sign in to comment
Sign in to answer