How to upload CSV file of size more than 250MB to SharePoint in Office 365 Using Python

Question

I'm automating the upload of CSV files from our database to the SharePoint Document Library using the Office365-REST-Python-Client Python Library and the SharePoint App. I've granted complete access to a generic ID, which was then used to generate a Client ID/Secret that will allow me to connect to the SharePoint site from Python.

`from office365.runtime.auth.client_credential import ClientCredential from office365.sharepoint.client_context import ClientContext

context = ClientContext(SITE_URL).with_credentials(CID, SEC) target_folder = context.web.get_folder_by_server_relative_url(f"Shared Documents/{TARGET_FOLDER}") target_folder.upload_file(os.path.basename(ROOT_DIR + "//" + f_name), file_content).execute_query() `

However, I am prompted by "Microsoft.SharePoint.Client.InvalidClientQueryException', 'The request message is too big. The server does not allow messages larger than 262144000 bytes" ERROR if my CSV file size goes beyond 250 MB.

Could you please help me with an alternate way to achieve this requirement of mine?

Answer 1

Hi @Raj Pandit

Thanks for your time in advance ,if you have any issue about SharePoint, you are welcome to raise a ticket in this forum.

By the way, since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others." and according to the scenario introduced here: Answering your own questions on Microsoft Q&A, I would make a brief summary of this thread:

How to upload CSV file of size more than 250MB to SharePoint in Office 365 Using Python

Issue Symptom:

Failed to upload large files more than 250MB to SharePoint by Office365-REST-Python-Client

Solution

Upload large file by following code

import os

from office365.sharepoint.client_context import ClientContext
from tests import test_user_credentials, test_team_site_url

ctx = ClientContext(test_team_site_url).with_credentials(test_user_credentials)

target_url = "/sites/team/Shared Documents"
target_folder = ctx.web.get_folder_by_server_relative_url(target_url)
size_chunk = 1000000
# local_path = "../../../tests/data/big_buck_bunny.mp4"
local_path = "../../../tests/data/SharePoint User Guide.docx"


def print_upload_progress(offset):
    file_size = os.path.getsize(local_path)
    print("Uploaded '{0}' bytes from '{1}'...[{2}%]".format(offset, file_size, round(offset / file_size * 100, 2)))


with open(local_path, 'rb') as f:
    uploaded_file = target_folder.files.create_upload_session(f, size_chunk,
                                                              print_upload_progress).execute_query()

#uploaded_file = target_folder.files.create_upload_session(local_path, size_chunk, print_upload_progress).execute_query()
print('File {0} has been uploaded successfully'.format(uploaded_file.serverRelativeUrl))

More sample for reference

https://github.com/vgrem/Office365-REST-Python-Client/tree/master/examples/sharepoint/files

---

You could click the "Accept Answer" button for this summary to close this thread, and this can make it easier for other community member's to see the useful information when reading this thread. Thanks for your understanding!

Answer 2

This is solved; please ignore this post. I didn't find an option to delete a post; therefore, I'm writing this here.

Thank you for reading this so far.