Verification and Validation of Chat completion API and Fine tuning

Question

Verification and Validation of Chat completion API and Fine tuning

Seven Delta 0

I’m using Azure cognitive search configured on own data and using chat completion API to ask prompts for questions on the data stored in Azure BLOB storage.

How can the prompt returned by chat completion API be verified and validated as the prompt does not return any of the confidence scores? Is there any alternative method to do this?

In addition is fine tuning of the gpt-3.5-turbo effective? Does it help better the responses on the data given in the BLOB storage? Please suggest and advise.

YutongTie-MSFT 36,821 Reputation points

2023-09-25T22:57:00.4233333+00:00

Hello @Seven Delta

Thanks for reaching out to us. The prompt returned by the chat completion API can be verified and validated by comparing it to the original prompt that was provided to the API. You can also compare the response to the prompt to the expected response based on the context of the conversation. While the API does not provide confidence scores, you can use these methods to evaluate the quality of the response.

You can implement your own methods for verifying and validating prompts by following these steps:

a. Control Prompts: When using the API, you can provide explicit instructions or control tokens to guide the model's behavior. You can start your prompt with specific instructions or context that make your request more explicit.

b. Response Filtering: After receiving a response from the API, you can filter and validate it based on predefined criteria. For example, you can check if the response is coherent, relevant, and accurate to your query.

c. User Feedback: Collect user feedback on the responses. If users find issues with the answers or prompts, you can use this feedback to improve and validate the prompts over time.

d. Human Review: Implement a process for human review to validate responses. Some responses may require manual validation, especially if they involve sensitive or critical information.

e. Iterative Improvement: Continuously refine your prompts and instructions based on feedback and validation results to improve the quality of responses over time.

Regarding fine-tuning the GPT-3.5-turbo model, it can be effective in improving the responses on the data given in the BLOB storage. Fine-tuning allows you to train the model on your specific data and context, which can improve the accuracy and relevance of the responses. However, fine-tuning requires a significant amount of data and computational resources, so it may not be feasible for all use cases. You should evaluate the benefits and costs of fine-tuning for your specific application.

Here's how you can approach fine-tuning:

a. Dataset Preparation: Prepare a dataset that is relevant to your task. In your case, if you're working with data from Azure BLOB storage, you would want to curate a dataset that includes similar content or questions related to your data.

b. Training: Fine-tune the GPT-3.5 Turbo model using your custom dataset. You can use platforms like OpenAI's fine-tuning services or libraries like Hugging Face Transformers to perform the fine-tuning.

c. Evaluation: After fine-tuning, evaluate the model's performance using a separate validation dataset. This helps ensure that fine-tuning has improved the model's capabilities for your specific task.

d. Iteration: Fine-tuning is often an iterative process. You may need to experiment with different hyperparameters, architectures, and dataset sizes to achieve the desired level of performance.

e. Monitoring: Continuously monitor the fine-tuned model's performance in your production environment and make adjustments as needed.

It's worth noting that while fine-tuning can be effective, GPT-3.5 Turbo is a highly capable model even without fine-tuning. Before deciding to fine-tune, it's advisable to test the base model to see if it meets your requirements.

I hope this helps!

Regards,

Yutong

Verification and Validation of Chat completion API and Fine tuning

Activity