Customize model responses to function outputs is not working due to 'function' role validation error

Question

Customize model responses to function outputs is not working due to 'function' role validation error

Hong Yong Ching 0

From the February news, it stated that fine-tuning with function calling is now supported. I tried to follow the docs and import the sample data by following the document, which is not working at all.

{
    "messages": [
        {"role": "user", "content": "What is the weather in San Francisco?"},
        {"role": "assistant", "function_call": {"name": "get_current_weather", "arguments": "{\"location\": \"San Francisco, USA\", \"format\": \"celcius\"}"}}
        {"role": "function", "name": "get_current_weather", "content": "21.0"},
        {"role": "assistant", "content": "It is 21 degrees celsius in San Francisco, CA"}
    ],
    "functions": [...] // same as before
}

Currently importing data consisting role function causing the following error.

Validation of jsonl file failed: Role must be either 'user', 'assistant', or 'system'. Unexpected value encountered on line 0. Please fix the data and try again.
For more details, refer to the documentation at: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython&pivots=programming-language-python#prepare-your-training-and-validation-data

remove or simply change the function role from below line to 'user', 'assistant', or 'system' works very well, but it is not how the function calling work.

 {"role": "function", "name": "get_current_weather", "content": "21.0"}

Is the fine-tuning on function calling response officially supported? Or am I getting anything wrong here?

navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-26T04:30:00.7133333+00:00

@Hong Yong Ching Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. The replacement for functions is the tools parameter. The replacement for function_call is the tool_choice parameter.

More Info here and here.

Suggestion:

Function calling is available in the 2023-07-01-preview API version and works with version 0613 of gpt-35-turbo, gpt-35-turbo-16k, gpt-4, and gpt-4-32k.

Sample code is available here:

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling?tabs=python#using-function-in-the-chat-completions-api-deprecated

Also note fine-tuning of some of the model is limited to a subset of regions, and is not available in every region the base model is available. Please confirm if your model is listed in the region mentioned in this article.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
Hong Yong Ching 0 Reputation points

2024-02-26T05:40:00.6333333+00:00

@navba-MSFT Thanks for reply! I know that function is deprecated in completion api, but the fine-tuning documents stated that we should use the legacy functions, and this is same as the OpenAI official fine-tuning that don't use tools parameter for fine-tuning too

From the doc caution: Important The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. However, the fine-tuning API currently requires use of the legacy parameters.

I'm fine-tuning gpt-3.5-turbo 1106 at north central US which works very well without the function role in messages. Again, the functions parameter work very well in fine-tuning, the problem is the function role in the messages list. it's not about completion api, nor the functions key.
Pinnela Madhuri 0 Reputation points

2024-02-26T05:44:34.8066667+00:00

[ { "role": "user", "content": "What is the weather in San Francisco?" }, { "role": "assistant", "content": "Let me check that for you." }, { "role": "assistant", "content": "The current weather in San Francisco, USA, is 21 degrees Celsius." } ]
Hong Yong Ching 0 Reputation points

2024-02-26T05:45:08.7833333+00:00

@navba-MSFT Thanks for reply!
Yes i understand that function is deprecated in completion api, but I'm talking about fine-tuning here**.** In fine-tuning documentations, it stated that we should use legacy function parameter as in OpenAI doc and also Azure doc. Cautions from Azure doc: Important The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. However, the fine-tuning API currently requires use of the legacy parameters. The functions parameter isn't the problem as fine-tuning still using legacy function way, the problem is the function role in messages list. Again, fine-tuning work very well without the function role and with the functions parameter. While the function role in messages is important for fine-tuning with function calling responses.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-26T05:51:29.69+00:00

@Hong Yong Ching Thanks for clarification. May I know which API-Version are you using ? Did you try using 2023-07-01-preview version ?

Also note that the training data files must be formatted as JSONL files, encoded in UTF-8 with a byte-order mark (BOM). The file must be less than 512 MB in size. Also Similar to training data files, validation data files must be formatted as JSONL files, encoded in UTF-8 with a byte-order mark (BOM). The file must be less than 100 MB in size. Could you please check if this requirement is fulfilled ?
Hong Yong Ching 0 Reputation points

2024-02-26T06:08:22.1466667+00:00
I'm using 2023-12-01-preview and nor 2023-07-01-preview working too. I don't think the API version is the issue because the error is raised on the stage of importing data to Azure OpenAI Studio before doing anything to model, and when we upload the training data through UI under Data Files section we are not allow to choose API version as well. While uploading data on the UI, it raised the same validation error too. About the data, I'm using the one provided in doc shown as below to test the issue:

{"messages": [{"role": "user", "content": "What is the weather in San Francisco?"}, {"role": "assistant", "function_call": {"name": "get_current_weather", "arguments": "{\"location\": \"San Francisco, USA\", \"format\": \"celcius\"}"}}, {"role": "function", "name": "get_current_weather", "content": "21.0"}, {"role": "assistant", "content": "It is 21 degrees celsius in San Francisco, CA"}], "functions": [...]}

As stated at beginning, everything goes well if I change the function role in this line to either system/user/assistant

{"role": "function", "name": "get_current_weather", "content": "21.0"}

but the function role is how it behave when call completion api. function role works well in completion api but not when importing data for fine-tuning.

In short, the error prompt by Azure is the validation error which I stated at the beginning. This is clearly the validation script when importing training data is not updated to support function role.

Validation of jsonl file failed: Role must be either 'user', 'assistant', or 'system'. Unexpected value encountered on line 0. Please fix the data and try again. For more details, refer to the documentation at: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython&pivots=programming-language-python#prepare-your-training-and-validation-data
Hong Yong Ching 0 Reputation points

2024-02-26T06:10:52.63+00:00

@Pinnela Madhuri Thanks for the reply, did u tried replacing function with assistant? I am afraid that this is abusing the model because when we are calling completion api, the input is actually using function role instead of assistant.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-29T04:50:44.61+00:00

@Hong Yong Ching I am discussing this internally with the Product Owners. I will get back once I have an update on this.
Hong Yong Ching 0 Reputation points

2024-03-01T01:08:42.2566667+00:00

@navba-MSFT Sure! Looking forward for good news from you!
Mi Sun Park 0 Reputation points Microsoft Employee

2024-03-01T18:13:35.0466667+00:00
@Hong Yong Ching, thank you for this report.

Wil fix it ASAP.

(as a temp solution) you can remove the "function" and the following "assistant" in the dataset. In fact, function calling is not about executing the function itself, it's about LLM to find the right function name and arguments given user content (the first two roles "user" and the following "assistant" in the message) And, after your app executing the function w/ the found function name/arguments from the LLM's first response, you pass in your function's response to the LLM again and LLM finally returns its 2nd response in user friendly way (the 3rd role "function" and the following 4th role "assistant" in the message). But, agree that it's good to have the 3rd and 4th roles in the dataset to teach LLM how to response back to users after it get the function response.
Hong Yong Ching 0 Reputation points

2024-03-04T01:56:04.65+00:00

@Mi Sun Park Thanks for the reply!

I understand the idea that LLM basically just generate these words with special token and trigger external API when captured, hence the function calling fine-tuning did worked half way until generating the func name and arguments.

However, my use case is focus on LLM extracting the right information from the API response, hence the "function" role is necessary.

Will wait for the fix, thanks!
Mi Sun Park 0 Reputation points Microsoft Employee

2024-03-07T03:36:54.6833333+00:00
@Hong Yong Ching , the fix is now available in North Central US and sweden central. Thank you for your patient.

fyi - we have a blog post about function calling. Fine Tuning with Function Calling on Azure OpenAI Service - Microsoft Community Hub
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-03-07T05:11:44.48+00:00

@Hong Yong Ching We appreciate your patience on this. I hope you had a chance to look at the above comment. Could you please test and check if you are encountering the same issue ? Awaiting your reply.
Hong Yong Ching 0 Reputation points

2024-03-15T03:54:57.5566667+00:00

@Mi Sun Park @navba-MSFT Hi! Sorry for the late reply! I could confirmed that the error is fixed! Thank you for the fast responding!

Just an additional inquiry, do we have any method to temporary disable deployed FT model other than deleting it?

Since when the FT model is deployed it cost the hosting fee, it is kind of troublesome that we must first deployed for development and remove after instead of temporary disable.

Your answer

navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-26T04:30:00.7133333+00:00

@Hong Yong Ching Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. The replacement for functions is the tools parameter. The replacement for function_call is the tool_choice parameter.

More Info here and here.

Suggestion:

Function calling is available in the 2023-07-01-preview API version and works with version 0613 of gpt-35-turbo, gpt-35-turbo-16k, gpt-4, and gpt-4-32k.

Sample code is available here:

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling?tabs=python#using-function-in-the-chat-completions-api-deprecated

Also note fine-tuning of some of the model is limited to a subset of regions, and is not available in every region the base model is available. Please confirm if your model is listed in the region mentioned in this article.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
Hong Yong Ching 0 Reputation points

2024-02-26T05:40:00.6333333+00:00

@navba-MSFT Thanks for reply! I know that function is deprecated in completion api, but the fine-tuning documents stated that we should use the legacy functions, and this is same as the OpenAI official fine-tuning that don't use tools parameter for fine-tuning too

From the doc caution: Important The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. However, the fine-tuning API currently requires use of the legacy parameters.

I'm fine-tuning gpt-3.5-turbo 1106 at north central US which works very well without the function role in messages. Again, the functions parameter work very well in fine-tuning, the problem is the function role in the messages list. it's not about completion api, nor the functions key.
Pinnela Madhuri 0 Reputation points

2024-02-26T05:44:34.8066667+00:00

[ { "role": "user", "content": "What is the weather in San Francisco?" }, { "role": "assistant", "content": "Let me check that for you." }, { "role": "assistant", "content": "The current weather in San Francisco, USA, is 21 degrees Celsius." } ]
Hong Yong Ching 0 Reputation points

2024-02-26T05:45:08.7833333+00:00

@navba-MSFT Thanks for reply!
Yes i understand that function is deprecated in completion api, but I'm talking about fine-tuning here**.** In fine-tuning documentations, it stated that we should use legacy function parameter as in OpenAI doc and also Azure doc. Cautions from Azure doc: Important The functions and function_call parameters have been deprecated with the release of the 2023-12-01-preview version of the API. However, the fine-tuning API currently requires use of the legacy parameters. The functions parameter isn't the problem as fine-tuning still using legacy function way, the problem is the function role in messages list. Again, fine-tuning work very well without the function role and with the functions parameter. While the function role in messages is important for fine-tuning with function calling responses.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-26T05:51:29.69+00:00

@Hong Yong Ching Thanks for clarification. May I know which API-Version are you using ? Did you try using 2023-07-01-preview version ?

Also note that the training data files must be formatted as JSONL files, encoded in UTF-8 with a byte-order mark (BOM). The file must be less than 512 MB in size. Also Similar to training data files, validation data files must be formatted as JSONL files, encoded in UTF-8 with a byte-order mark (BOM). The file must be less than 100 MB in size. Could you please check if this requirement is fulfilled ?
Hong Yong Ching 0 Reputation points

2024-02-26T06:08:22.1466667+00:00

I'm using 2023-12-01-preview and nor 2023-07-01-preview working too. I don't think the API version is the issue because the error is raised on the stage of importing data to Azure OpenAI Studio before doing anything to model, and when we upload the training data through UI under Data Files section we are not allow to choose API version as well. While uploading data on the UI, it raised the same validation error too. About the data, I'm using the one provided in doc shown as below to test the issue:

{"messages": [{"role": "user", "content": "What is the weather in San Francisco?"}, {"role": "assistant", "function_call": {"name": "get_current_weather", "arguments": "{\"location\": \"San Francisco, USA\", \"format\": \"celcius\"}"}}, {"role": "function", "name": "get_current_weather", "content": "21.0"}, {"role": "assistant", "content": "It is 21 degrees celsius in San Francisco, CA"}], "functions": [...]}

As stated at beginning, everything goes well if I change the function role in this line to either system/user/assistant

{"role": "function", "name": "get_current_weather", "content": "21.0"}

but the function role is how it behave when call completion api. function role works well in completion api but not when importing data for fine-tuning.

In short, the error prompt by Azure is the validation error which I stated at the beginning. This is clearly the validation script when importing training data is not updated to support function role.

Validation of jsonl file failed: Role must be either 'user', 'assistant', or 'system'. Unexpected value encountered on line 0. Please fix the data and try again. For more details, refer to the documentation at: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython&pivots=programming-language-python#prepare-your-training-and-validation-data
Hong Yong Ching 0 Reputation points

2024-02-26T06:10:52.63+00:00

@Pinnela Madhuri Thanks for the reply, did u tried replacing function with assistant? I am afraid that this is abusing the model because when we are calling completion api, the input is actually using function role instead of assistant.
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-02-29T04:50:44.61+00:00

@Hong Yong Ching I am discussing this internally with the Product Owners. I will get back once I have an update on this.
Hong Yong Ching 0 Reputation points

2024-03-01T01:08:42.2566667+00:00

@navba-MSFT Sure! Looking forward for good news from you!
Mi Sun Park 0 Reputation points Microsoft Employee

2024-03-01T18:13:35.0466667+00:00

@Hong Yong Ching, thank you for this report.

Wil fix it ASAP.

(as a temp solution) you can remove the "function" and the following "assistant" in the dataset. In fact, function calling is not about executing the function itself, it's about LLM to find the right function name and arguments given user content (the first two roles "user" and the following "assistant" in the message) And, after your app executing the function w/ the found function name/arguments from the LLM's first response, you pass in your function's response to the LLM again and LLM finally returns its 2nd response in user friendly way (the 3rd role "function" and the following 4th role "assistant" in the message). But, agree that it's good to have the 3rd and 4th roles in the dataset to teach LLM how to response back to users after it get the function response.
Hong Yong Ching 0 Reputation points

2024-03-04T01:56:04.65+00:00

@Mi Sun Park Thanks for the reply!

I understand the idea that LLM basically just generate these words with special token and trigger external API when captured, hence the function calling fine-tuning did worked half way until generating the func name and arguments.

However, my use case is focus on LLM extracting the right information from the API response, hence the "function" role is necessary.

Will wait for the fix, thanks!
Mi Sun Park 0 Reputation points Microsoft Employee

2024-03-07T03:36:54.6833333+00:00

@Hong Yong Ching , the fix is now available in North Central US and sweden central. Thank you for your patient.

fyi - we have a blog post about function calling. Fine Tuning with Function Calling on Azure OpenAI Service - Microsoft Community Hub
navba-MSFT 27,540 Reputation points Microsoft Employee Moderator

2024-03-07T05:11:44.48+00:00

@Hong Yong Ching We appreciate your patience on this. I hope you had a chance to look at the above comment. Could you please test and check if you are encountering the same issue ? Awaiting your reply.
Hong Yong Ching 0 Reputation points

2024-03-15T03:54:57.5566667+00:00

@Mi Sun Park @navba-MSFT Hi! Sorry for the late reply! I could confirmed that the error is fixed! Thank you for the fast responding!

Just an additional inquiry, do we have any method to temporary disable deployed FT model other than deleting it?

Since when the FT model is deployed it cost the hosting fee, it is kind of troublesome that we must first deployed for development and remove after instead of temporary disable.

Share via

Customize model responses to function outputs is not working due to 'function' role validation error

Your answer