How to use ADF to save daily data from Oracle

Question

How to use ADF to save daily data from Oracle

Lucas 0

Hello Everyone.

Here is the situation. We have an Oracle Database, now we want to move all the data from Oracle to Azure.

The Database refreshes daily(with no historical data).

We want to save the data in Data Lake Storage via ADF.

Is there any way that I can save daily data in one file(more like append the data)?

Please help,

Thank you

Dillon Silzer 60,701 Reputation points Volunteer Moderator

2023-06-29T04:29:14.7733333+00:00

Have you used a notebook such as Azure Databricks before? Do you have an APIs that you can use to extract the data from the Oracle Database? You could use ADF to spawn some Azure Databrick solution that queries your database and saves it in Data Lake Storage.

1 answer

Your answer

Dillon Silzer 60,701 Reputation points Volunteer Moderator

2023-06-29T04:29:14.7733333+00:00

Have you used a notebook such as Azure Databricks before? Do you have an APIs that you can use to extract the data from the Oracle Database? You could use ADF to spawn some Azure Databrick solution that queries your database and saves it in Data Lake Storage.

Answer 1

Hi Lucas ,

Welcome to Microsoft Q&A platform and thanks for posting your question here. As I understand your question , you want to load data on daily basis from oracle database to ADLS via ADF pipeline.

To move data from Oracle to Azure Data Lake Storage via ADF, you can use the Oracle connector in the source and the Azure Data Lake Storage Gen2 connector in the sink.

You can schedule the ADF pipeline using scheduled trigger to run daily to move the data from Oracle to Azure Data Lake Storage. User's image

Note: There is no way to append the data in a files in ADLS, it will overwrite the file everytime the pipeline will run. You can use table in sink to append the data or else , keep separate files for every day run and at the end of the week, run a pipeline which takes uses copy activity to merge all the files together into a single file.

Hope it helps. Kindly accept the answer if it helps. Thankyou.

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2023-07-03T06:20:22.43+00:00

Hi Lucas ,

Just checking in to see if the above answer helped. Please do consider clicking Accept Answer as accepted answers help community as well. Also, please click on Yes for the survey 'Was the answer helpful'

Share via

How to use ADF to save daily data from Oracle

1 answer

Your answer