How can I upload a csv from a local drive to Lakehouse programmatically, so I can schedule the pickup?

Conner, Kyle 20

Hi,

We are storing csvs on a local drive to support reporting and analysis. I would like to pull these csvs into the Lakehouse on a scheduled basis programmatically. All I can find so far is a manual upload. Can I code a Notebook, or leverage a Dataflow(Gen2) to pull the file into the Lakehouse, and have it scheduled as these files are updated in the source every day?

Only solutions I have seen is manual upload of csvs, and a Dataflow (Gen2) upload of Excel, neither will work to my knowledge.

Thanks,

Vinodh247-1375 11,396 Reputation points

2024-04-06T10:26:31.9866667+00:00

Just checking if you had a chance to look at this. if so, can you please update it?
Conner, Kyle 20 Reputation points

2024-04-06T12:29:46.6333333+00:00

Thanks for the answer. I'll keep looking online to try to find the detail I am looking for.

Accepted answer

Vinodh247-1375 11,396 Reputation points

2024-04-04T15:16:54.83+00:00

Hi Conner, Kyle,

Thanks for reaching out to Microsoft Q&A.

Can I code a Notebook, or leverage a Dataflow(Gen2) to pull the file into the Lakehouse, and have it scheduled as these files are updated in the source every day?

Yes, you can, both options will work. Set the pipeline to run daily at a specific time to keep your Lakehouse data up-to-date.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.
Conner, Kyle 20 Reputation points

2024-04-04T15:21:29.6566667+00:00

Can you point me to an example? I have been unable to find an example on the Internet at this point.

Vinodh247-1375 11,396 Reputation points

2024-04-05T09:16:21.75+00:00

Dataflows: Create a new Dataflow Gen2 in your workspace. Add a source that points to your files (CSV, Parquet, etc.). Configure the data destination to be your Lakehouse. Specify the update method (ex: replace/append). Publish the dataflow & set up a pipeline that runs this dataflow on a daily schedule.

Scheduled Refresh using Pipelines: Create a pipeline that triggers your dataflow. Configure the pipeline to run daily at a specific time. In the pipeline, add an activity that runs your dataflow. Specify the source and destination for your files. Set the refresh frequency to daily.

Notebooks with Spark Code: Write custom Spark code in a notebook. Connect to your data source (using JDBC, REST API, etc.). Load the files into Spark DataFrames. Save the DataFrames into your Lakehouse (ex: Delta tables). Schedule the notebook execution using ADF or other scheduling tools.
Sign in to comment

Share via

How can I upload a csv from a local drive to Lakehouse programmatically, so I can schedule the pickup?

0 additional answers