Can I use ChatTokenLogProbability LogProb as a confidence score for a message?

Question

By ChatTokenLogProbability I'm specifically talking about the documentation on this page:

https://learn.microsoft.com/en-us/javascript/api/%40azure/openai/chattokenlogprobabilityresult?view=azure-node-preview

It describes logprob as the probability of a token from the message. I was theorizing about asking my model to rank the similarity of a list of words to one word. If the model responds with a listing could I in theory look at the log probability of the tokens that make up each word and use it as a confidence score?

An example prompt could be "rank the similarity of 'purple' to the following words: 'truck', 'water', 'red', 'cup'. respond only with a list of the words in the order of most similar to least similar". Then could I look at the probabilities as a confidence of putting each specific word in its position on the list?

Quick addition: The link above shows the documentation in JavaScript. Is this also supported in Python? What is the syntax for it if so? Also if anyone has recommendations of a better confidence scoring system using OpenAI let me know.

Answer 1

Kayla Farivar Greetings!

I haven't found any python examples except this. Did you see these documentations Confidence score and Improve quality of response with synonyms already?

Regarding your question about a better confidence scoring system using OpenAI, it really depends on your specific use case and the type of data you're working with. However, you can experiment with different models and parameters to see which one works best for your needs.