How to upload CSV file of size more than 250MB to SharePoint in Office 365 Using Python

Raj Pandit 20 Reputation points
2023-02-26T09:53:04.47+00:00

I'm automating the upload of CSV files from our database to the SharePoint Document Library using the Office365-REST-Python-Client Python Library and the SharePoint App. I've granted complete access to a generic ID, which was then used to generate a Client ID/Secret that will allow me to connect to the SharePoint site from Python.

`from office365.runtime.auth.client_credential import ClientCredential from office365.sharepoint.client_context import ClientContext

context = ClientContext(SITE_URL).with_credentials(CID, SEC) target_folder = context.web.get_folder_by_server_relative_url(f"Shared Documents/{TARGET_FOLDER}") target_folder.upload_file(os.path.basename(ROOT_DIR + "//" + f_name), file_content).execute_query() `

However, I am prompted by "Microsoft.SharePoint.Client.InvalidClientQueryException', 'The request message is too big. The server does not allow messages larger than 262144000 bytes" ERROR if my CSV file size goes beyond 250 MB.

Could you please help me with an alternate way to achieve this requirement of mine?

SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,276 questions
SharePoint Development
SharePoint Development
SharePoint: A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.Development: The process of researching, productizing, and refining new or existing technologies.
2,807 questions
0 comments No comments
{count} votes

Accepted answer
  1. RaytheonXie_MSFT 33,641 Reputation points Microsoft Vendor
    2023-02-27T05:28:59.28+00:00

    Hi @Raj Pandit

    Thanks for your time in advance ,if you have any issue about SharePoint, you are welcome to raise a ticket in this forum.

    By the way, since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others." and according to the scenario introduced here: Answering your own questions on Microsoft Q&A, I would make a brief summary of this thread:

    How to upload CSV file of size more than 250MB to SharePoint in Office 365 Using Python

    Issue Symptom:

    Failed to upload large files more than 250MB to SharePoint by Office365-REST-Python-Client

    Solution

    Upload large file by following code

    
    import os
    
    from office365.sharepoint.client_context import ClientContext
    from tests import test_user_credentials, test_team_site_url
    
    ctx = ClientContext(test_team_site_url).with_credentials(test_user_credentials)
    
    target_url = "/sites/team/Shared Documents"
    target_folder = ctx.web.get_folder_by_server_relative_url(target_url)
    size_chunk = 1000000
    # local_path = "../../../tests/data/big_buck_bunny.mp4"
    local_path = "../../../tests/data/SharePoint User Guide.docx"
    
    
    def print_upload_progress(offset):
        file_size = os.path.getsize(local_path)
        print("Uploaded '{0}' bytes from '{1}'...[{2}%]".format(offset, file_size, round(offset / file_size * 100, 2)))
    
    
    with open(local_path, 'rb') as f:
        uploaded_file = target_folder.files.create_upload_session(f, size_chunk,
                                                                  print_upload_progress).execute_query()
    
    #uploaded_file = target_folder.files.create_upload_session(local_path, size_chunk, print_upload_progress).execute_query()
    print('File {0} has been uploaded successfully'.format(uploaded_file.serverRelativeUrl))
    
    

    More sample for reference

    https://github.com/vgrem/Office365-REST-Python-Client/tree/master/examples/sharepoint/files


    You could click the "Accept Answer" button for this summary to close this thread, and this can make it easier for other community member's to see the useful information when reading this thread. Thanks for your understanding!

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. Raj Pandit 20 Reputation points
    2023-02-26T10:48:38.6+00:00

    This is solved; please ignore this post. I didn't find an option to delete a post; therefore, I'm writing this here.

    Thank you for reading this so far.