Azure OpenAI:Testing and training internal error when uploading files

Chilberto 231

I have a newly provisioned OpenAI in East US. I am trying to upload a .jsonl file in File Management. I tried several attempts all resulting in "internalFailure: Internal failure."

My last attempts were using the examples:

{
    "prompt":"Item is a handbag. Colour is army green. Price is midrange. Size is small.->",
    "completion":"This stylish small green handbag will add a unique touch to your look, without costing you a fortune."
}

Are the examples snippets or complete examples?

My guess is there is an encoding issue or something subtle. I am using notepad++ to create.

romungi-MSFT 43,681 Reputation points Microsoft Employee

2023-03-30T11:38:09.4333333+00:00
Chilberto I have used the data preparation tool with your input and appended it to my file and uploaded. It seems to have processed successfully. I have used it in the following format in my file and the data preparation tool only recommended to add a whitespace at the beginning of completion. Please see the content of the file used below:

{"prompt":"When I go to the store test, I want an ->","completion":" apple."} {"prompt":"When I go to work, I want a ->","completion":" coffee."} {"prompt":"When I go home, I want a ->","completion":" soda."} {"prompt":"Item is a handbag. Colour is army green. Price is midrange. Size is small.->","completion":"This stylish small green handbag will add a unique touch to your look, without costing you a fortune."}

The file was successfully uploaded. It worked fine with a resource in East US and South Central US. Please try to use the tool and format as above to check if it works for you. Thanks!!
Chilberto 231 Reputation points

2023-03-30T20:46:02.7233333+00:00

This is what I did. I took your input above using the Copy feature and pasted in Notepad++ ensuring the encoding was UTF-8. I then uploaded using the Upload a new dataset. I repeated this for the other post's sample also.

I received the same error during processing: "internalFailure: Internal failure."

I then provisioned another OpenAI instance and repeated the steps with the same file.

Both new files processed successfully. I then posted my original file. The file did fail but with a polite message:

Validation of jsonl file failed: No valid json line found in file. Each line's json may only contain two properties: ("context" or "prompt") and "completion".

I am just starting my journey so will not lose any work by switching to the new instance. I have seen one other post describing the same behaviour. Perhaps there is an issue during provisioning of the service that the MS team have not worked out yet.

Thanks for your help.
Guy Elsmore-Paddock 6 Reputation points

2023-04-26T00:14:34.57+00:00
It appears that the validator is broken for "open-ended" training. There are several tickets in the docs project that point out that the validator will not accept a training data set that has empty prompts:

https://github.com/MicrosoftDocs/azure-docs/issues/107548

https://github.com/MicrosoftDocs/azure-docs/issues/108515

romungi-MSFT 43,681 Microsoft Employee

Guy Elsmore-Paddock I tried to use an open ended file recently following the guidelines shared and it has accepted the file. Using only "open ended" entries in the complete file though fails the processing of the file. Here are the entries I used in my file.

{"prompt":"When I go to the store, I want an ->\n\n###\n\n","completion":" apple\n."}
{"prompt":"When I go to work, I want a ->\n\n###\n\n","completion":" coffee\n."}
{"prompt":"When I go home, I want a ->\n\n###\n\n","completion":" soda\n."}
{"prompt":"Company: BHFF insurance\nProduct: allround insurance\nAd:One stop shop for all your insurance needs!\nSupported:\n\n###\n\n","completion":" yes."}
{"prompt":"Company: Loft conversion specialists\nProduct: -\nAd:Straight teeth in weeks!\nSupported:\n\n###\n\n","completion":" no."}
{"prompt":"","completion":" Ya Sure"}
{"prompt":"","completion":" Let me test this!!"}

User's image

Accepted answer

Erkan Sahin 830 Reputation points

2023-03-29T22:50:47.6633333+00:00
The example you provided seems to be a snippet of data rather than a complete file.

To create a properly formatted .jsonl file, you should ensure that each line in the file is a valid JSON object, and the objects are separated by newlines. Here's a simple example of a valid .jsonl file:

{"prompt": "Item is a handbag. Colour is army green. Price is midrange. Size is small.", "completion": "This stylish small green handbag will add a unique touch to your look, without costing you a fortune."} {"prompt": "Item is a watch. Colour is black. Price is high-end. Size is medium.", "completion": "This sleek black watch is the perfect accessory for those who value both style and quality."}

When creating the .jsonl file using Notepad++, you can ensure that you are using the correct encoding by following these steps:

Click on "Encoding" in the top menu.

Select "Encode in UTF-8" (without BOM) if it's not already selected.

Additionally, you can validate your .jsonl file using online tools like JSONL Validator to make sure there are no errors in the file's format.

If you've checked the formatting and encoding of your .jsonl file and the issue persists, you might want to contact OpenAI support for further assistance. They should be able to provide more detailed guidance on resolving the "internalFailure" error.
Please sign in to rate this answer.

1 person found this answer helpful.
Chilberto 231 Reputation points

2023-03-30T20:49:29.53+00:00

Thanks Erkan. I did the following to resolve. I copied your example and put in a new file making sure the encoding was UTF-8. I then uploaded with the same result.

I then provisioned a new instance of OpenAI. In the new instance I took you sample and it processed without issue. I then took my original file and it failed but with a polite error:

Validation of jsonl file failed: No valid json line found in file. Each line's json may only contain two properties: ("context" or "prompt") and "completion".

I have seen another post describing the same behaviour. As I am just starting this has little impact so I will switch to the new instance. Thanks for your help.
Sign in to comment

Share via

Azure OpenAI:Testing and training internal error when uploading files

0 additional answers