How can I increase the filesize-limit of 500MB for training data for fine-tuning in Azure OpenAI?

Hauke Kraemer 20 Reputation points
2024-09-19T19:18:46.3233333+00:00

When trying to fine-tune a base model via Azure OpenAI Studio I stumbled across the filesize-limit of 200MB for direct upload and 500MB for uploading through a blob, respectively.

Screenshot 2024-09-19 at 21.12.15

Screenshot 2024-09-19 at 21.12.24

Splitting my training data into chunks meeting these limits also does not help, since I can only pick a single data file of all my uploaded data files. This is very unfortunate. How can I increase this limit? Would it help to create the fine-tuning-job differently, e.g. Python SDK?

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,052 questions
{count} votes

Accepted answer
  1. AshokPeddakotla-MSFT 33,761 Reputation points
    2024-09-20T03:59:29.26+00:00

    Hauke Kraemer Greetings & Welcome to Microsoft Q&A forum!

    Your understanding is correct regarding the limits.

    Training data files must be formatted as JSONL files, encoded in UTF-8 with a byte-order mark (BOM). The file must be less than 512 MB in size.

    Splitting my training data into chunks meeting these limits also does not help, since I can only pick a single data file of all my uploaded data files. This is very unfortunate. How can I increase this limit? Would it help to create the fine-tuning-job differently, e.g. Python SDK?

    One thing you could try is to submit a service request for increasing the limits.

    Do let me know if you have any further queries.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.