Self-train OpenAI model using response and feedback

Question

using openAI services (GPT model 3.5) How can I train the module, while asking questions

(by using user feedback model should be trained)

Answer 1

Hello @Sajith Hettiarachchige

Thanks for reaching out to us, are you mentioning you want to retrain your fine-tuned model again with more feedbacks? Please correct me if my understanding is not correct.

As I have mentioned in your another thread -

There is currently no automatically way to continue fine-tuning the model.

You need to build a process by yourself and include the continuing fine-tuning as part of it.

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython-new&pivots=programming-language-python#continuous-fine-tuning

An example of this process is as below -

Collect Feedback

Implement Feedback Collection

• Integrate Feedback Mechanism: Include a feedback option in your application where users can rate or provide comments on the model’s responses.
• Store Feedback: Collect and store feedback in a structured format. For example, you could use a database or cloud storage (like Azure SQL Database or Azure Blob Storage) to save feedback data.

Analyze and Prepare Feedback

Data Aggregation

• Collect Responses and Feedback: Gather all interactions, including the original model responses and user feedback.
• Label Data: Depending on your feedback, label the data accordingly. For example, if feedback is positive or negative, or if it provides suggestions for improvement.

Data Cleaning

• Filter Feedback: Remove noisy or irrelevant feedback. Ensure that the feedback is useful for training.
• Format Data: Convert feedback into a format suitable for training, such as pairs of prompts and improved responses.

Prepare Training Data

Data Annotation

• Create Training Examples: Prepare training examples from feedback, such as prompt-response pairs with corrected or improved responses.
• Organize Data: Structure the data to be used for fine-tuning. Ensure consistency and quality.

Automate Model Training

Use OpenAI’s Fine-Tuning API (or equivalent in Azure OpenAI)

• Prepare Training Scripts: Write scripts to prepare your data for fine-tuning.
• Fine-Tuning Process:
  • Create a Dataset: Upload your training data to the OpenAI or Azure OpenAI fine-tuning platform.

     - **Initiate Fine-Tuning:** Use the fine-tuning API to train your model on the new data.
    
              - **Monitor Training:** Track the training process to ensure it completes successfully.
    

Automate Training Pipelines

• Create Automation Scripts: Develop scripts or workflows to automate the process of collecting feedback, preparing data, and triggering fine-tuning.
• Set Up Scheduling: Use Azure services like Azure Functions or Logic Apps to schedule and automate these processes based on feedback accumulation.

Deploy and Test Updated Model

Deploy the Fine-Tuned Model

• Deployment: Deploy the newly trained model to your production environment.
• Testing: Perform thorough testing to ensure the updated model performs as expected.

Continuous Monitoring

• Monitor Performance: Continuously monitor the performance of the model to ensure improvements.
• Adjust Training: Based on performance, adjust your training process and data preparation as needed.

Please adjust this process accordingly. I hope this helps.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.