Copy the files from dbfs path in databricks to SharePoint location

Question

Copy the files from dbfs path in databricks to SharePoint location

AZ_Krishna Seshadry Gadepalli 20

I have a databricks code to load the data into dbfs path from the dataframe looking for the python code where I need to copy the files available in the dbfs path to sharepoint location folder so that users can download the csv file extracts

AZ_Krishna Seshadry Gadepalli 20 Reputation points

2024-02-12T12:32:29.7266667+00:00

Hi @ShaikMaheer-MSFT could you please help me with the above question In the databricks we are loading the data into dbfs path from the data frame and sharing the extract manually by creating a view or copying the data from dbfs path to local and then sharing the extract. we are looking to automate it by copying the files placed in the dbfs path to the SharePoint location where users can download it from the SharePoint location .

AZ_Krishna Seshadry Gadepalli 20

Hi the issue is fixed till the point where directory is created in sharepoint but the code to copy the csv file from dbfs path is not working now but the code is executing used the below code could you please help me what could be the issue

when I am running the databricks notebook I am storing the csv file into the dbfs path dbfs:/FileStore/tables/test/test_extract.csv

import os

def upload_to_sharepoint(sp_filepath, blob_file_path):
    blobpath = '/dbfs' + blob_file_path
    filename = os.path.basename(blob_file_path)
        # read content of file
    with open(blobpath, 'rb') as content_file:
        file_content = content_file.read()

    sp_relative_url = create_sharepoint_directory(sp_filepath)
    ctx = get_sharepoint_context_using_app()

    target_folder = ctx.web.get_folder_by_server_relative_url(sp_relative_url)
    target_folder.upload_file(filename, file_content).execute_query()    # upload the file
    print('Uploaded ' + filename + ' to ' + sp_relative_url)

Accepted answer

0 additional answers

Your answer

AZ_Krishna Seshadry Gadepalli 20 Reputation points

2024-02-12T12:32:29.7266667+00:00

Hi @ShaikMaheer-MSFT could you please help me with the above question In the databricks we are loading the data into dbfs path from the data frame and sharing the extract manually by creating a view or copying the data from dbfs path to local and then sharing the extract. we are looking to automate it by copying the files placed in the dbfs path to the SharePoint location where users can download it from the SharePoint location .
AZ_Krishna Seshadry Gadepalli 20 Reputation points

2024-02-13T10:19:40.9133333+00:00

Hi the issue is fixed till the point where directory is created in sharepoint but the code to copy the csv file from dbfs path is not working now but the code is executing used the below code could you please help me what could be the issue

when I am running the databricks notebook I am storing the csv file into the dbfs path dbfs:/FileStore/tables/test/test_extract.csv

import os def upload_to_sharepoint(sp_filepath, blob_file_path): blobpath = '/dbfs' + blob_file_path filename = os.path.basename(blob_file_path) # read content of file with open(blobpath, 'rb') as content_file: file_content = content_file.read() sp_relative_url = create_sharepoint_directory(sp_filepath) ctx = get_sharepoint_context_using_app() target_folder = ctx.web.get_folder_by_server_relative_url(sp_relative_url) target_folder.upload_file(filename, file_content).execute_query() # upload the file print('Uploaded ' + filename + ' to ' + sp_relative_url)

Answer 1

Hi AZ_Krishna Seshadry Gadepalli, Thank you for posting query in Microsoft Q&A Platform.

You need to consider using Office365 SDKs for Python, which will help you to work with Sharepoint. > Consider writing python code to read files from dbfs and them load them to Sharepoint sites either by using Python SDKs or REST API calls in Python to Sharepoint.

Below link has sample example of using Office365 SDK in Python for uploading file to sharepoint. https://plainenglish.io/blog/how-to-upload-files-to-sharepoint-using-python Hope this helps. Please let me know how it goes.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Share via

Copy the files from dbfs path in databricks to SharePoint location

0 additional answers

Your answer