Data Integration from external MySQL database to Azure Databricks using Azure Data Factory

Question

Hi there,

I'm Márcio Antônio, I'm building a cloud environment to train my colleagues and having some difficulty using Azure Data Factory.
As the title suggests, I need to import data from an external MySQL database into my Databricks workspace.
I managed to link both services, MySQL and Databricks, however, I don't know how to send data from one to another.

I tried to use the "Copy data" service but there is no sink option pointing to my Databricks workspace.
I need to create only one copy of my data (from MySQL) but I don't want to send the data to a blob storage apart from DBFS.
(Recently I discovered that I need to create a new blob storage to be able to connect in Azure Data Factory, is that true?).
How do I do that?

My goal is to build this simple pipeline using Azure Data Factory for training purposes, What is the best approach it?

Answer 1

Hello @Márcio Antônio Freitas Júnior ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is to know how can you query the mysql from azure databicks , please do let us know if its not accurate.
Azure datebricks does provide compute power for data computation . It does not provide any storage mechanism as such apart from DBFS .
The below scipts when you run from a notebook can help connect to the mysql . From ADF also you can run the notebook by using the notebook activity .

driver = "org.mariadb.jdbc.Driver"

table = "authors"
user = "username@servername"
password = "password"

url = "jdbc:mysql://servername.mysql.database.azure.com:3306/dbname"

remote_table = (spark.read
.format("jdbc")
.option("driver", driver)
.option("url", url)
.option("dbtable", table)
.option("user", user)
.option("password", password)
.load()
)

Read more here : https://docs.databricks.com/external-data/mysql.html

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators