How to Retrieve Action/Plan/Reasoning with OpenaAI Assistant

Question

How to Retrieve Action/Plan/Reasoning with OpenaAI Assistant

Mayank Goel 0

I am creating azure ai assistant with below steps:

assistant = client.beta.assistants.create()

thread = client.beta.threads.create()

message = client.beta.threads.messages.create()

run = client.beta.threads.runs.create()

#Retrieve run
run = client.beta.threads.runs.retrieve()

#Retrieve Messages
messages = client.beta.threads.messages.list(thread_id=thread.id)

However, from messages/run object, I was curious to understand how agent/LLM are coming up with plan/action/reasoning and was not able to get any parameters.

Could someone guide me on how to properly retrieve these logs using the Python? Are there specific endpoints or configurations required to achieve this?

Any help or pointers would be greatly appreciated!

navba-MSFT 27,485 Microsoft Employee

@Mayank Goel Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

To add logging functionality to capture run details and message responses, including reasoning, you can follow either Approach 1 OR Approach 2.
.
.

Approach 1: You can have a custom logging functionality in your code:

import os

import logging

from azure.identity import DefaultAzureCredential, get_bearer_token_provider

from openai import AzureOpenAI

# Configure logging

logging.basicConfig(level=logging.INFO)

logger = logging.getLogger(__name__)

# Initialize Azure OpenAI client

token_provider = get_bearer_token_provider(DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default")

client = AzureOpenAI(

    azure_ad_token_provider=token_provider,

    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],

    api_version="2024-05-01-preview",

)

# Create an assistant

assistant = client.beta.assistants.create(

    name="Math Assist",

    instructions="You are an AI assistant that can write code to help answer math questions.",

    tools=[{"type": "code_interpreter"}],

    model="gpt-4-1106-preview" # Replace this with your model deployment name

)

# Create a thread

thread = client.beta.threads.create()

# Add a user question to the thread

message = client.beta.threads.messages.create(

    thread_id=thread.id,

    role="user",

    content="I need to solve the equation `3x + 11 = 14`. Can you help me?"

)

# Run the thread and poll for the result

run = client.beta.threads.runs.create_and_poll(

    thread_id=thread.id,

    assistant_id=assistant.id,

    instructions="Please address the user as Jane Doe. The user has a premium account.",

)

# Log the run status

logger.info(f"Run completed with status: {run.status}")

if run.status == "completed":

    # Retrieve and log messages

    messages = client.beta.threads.messages.list(thread_id=thread.id)

    for message in messages:

        logger.info(f"Message: {message.to_json(indent=2)}")

    # Log detailed response and reasoning

    if 'response' in run and 'reasoning' in run['response']:

        logger.info(f"Assistant reasoning: {run['response']['reasoning']}")

# Save logs to a file

logging.basicConfig(filename='assistant_logs.log', level=logging.INFO)

logging.info("Run completed with status: %s", run.status)

if run.status == "completed" and 'response' in run and 'reasoning' in run['response']:

    logging.info("Assistant run completed with reasoning: %s", run['response']['reasoning'])

.

Approach 2:

Invoking the REST API from your code to fetch the details about your run:

You can perform a List runs API call which would return a list of run step objects as shown here

From the response, you can examine Run Steps that allows you to understand how the Assistant is getting to its final results and check for the status.

Run status definitions

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant#run-status-definitions

in_progressWhile in_progress, the Assistant uses the model and tools to perform steps. You can view progress being made by the Run by examining the Run Steps.2. You can then invoke List run steps API which would Return a list of steps belonging to a run.

3. You can then invoke the Retrieve run . This returns The run object matching the specified run ID.

4. To Retrieve run step you can call the Retrieve run step API . This returns the run step object matching the specified ID.

.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

**

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

navba-MSFT 27,485 Reputation points Microsoft Employee

2024-12-09T04:24:10.7433333+00:00

@Mayank Goel Just following up to check if the below answer helped. If that answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let me know. I would be happy to help.
Mayank Goel 0 Reputation points

2024-12-09T21:53:57.5733333+00:00
I tried both the approaches and I am attaching run object here.Sample_Run.txt From here, we can see that run object doesn't have response/reasoning hence it is never giving any action/plan and below logging code doesn't yield anything:

if 'response' in run and 'reasoning' in run['response']:

logger.info(f"Assistant reasoning: {run['response']['reasoning']}") ```All I can see is there is response_format='auto' however no parameter related to plan/action/reasoning in run object at least. Even the message object doesn't have any action/plan/reasoning object/parameter. Can you please look into this and help?

navba-MSFT 27,485 Microsoft Employee

@Mayank Goel Thanks for getting back. I am sharing the details of my run response object:

.

Response details:

{ "id": "run_BnQFE1M3idPlxjI6H46DB6l6", "object": "thread.run", "created_at": 1731946971, "assistant_id": "asst_3rY50WWj2cvOXU5jjealB7qy", "thread_id": "thread_ePJeuEUaUaWM6Mos8CA0iNUo", "status": "incomplete", "started_at": 1731946972, "expires_at": null, "cancelled_at": null, "failed_at": null, "completed_at": 1731946990, "required_action": null, "last_error": null, "model": "gpt-4o", "instructions": "You are an AI assistant", "tools": [ { "type": "code_interpreter" }, { "type": "file_search", "file_search": { "ranking_options": { "ranker": "default_2024_08_21", "score_threshold": 0 } } } ], "tool_resources": {}, "metadata": {}, "temperature": 1, "top_p": 0.94, "max_completion_tokens": null, "max_prompt_tokens": null, "truncation_strategy": { "type": "auto", "last_messages": null }, "incomplete_details": { "reason": "max_prompt_tokens" }, "usage": { "prompt_tokens": 31293, "completion_tokens": 181, "total_tokens": 31474 }, "response_format": "auto", "tool_choice": "auto", "parallel_tool_calls": true }

You can see the status and reason attributes. Depending on the above structure, you can update the code to log this info:

.

# Create a thread
thread = client.beta.threads.create()

# Add a user question to the thread
message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="I need to solve the equation `3x + 11 = 14`. Can you help me?"
)

# Run the thread and poll for the result
run = client.beta.threads.runs.create_and_poll(
    thread_id=thread.id,
    assistant_id=assistant.id,
    instructions="Please address the user as Jane Doe. The user has a premium account."
)

# Add a delay of 30 seconds
time.sleep(30)

# Log the run status and other details
logger.info(f"Run completed with status: {run.status}")
logger.info(f"Run details: {run.to_json(indent=2)}")

# Log additional run details if available
if run.status == "incomplete":
    logger.warning(f"Run incomplete, reason: {run.incomplete_details.get('reason', 'No reason provided')}")

if run.status == "completed":
    # Retrieve and log messages
    messages = client.beta.threads.messages.list(thread_id=thread.id)
    for message in messages:
        logger.info(f"Message: {message.to_json(indent=2)}")
        if 'metadata' in message:
            logger.info(f"Metadata: {message['metadata']}")

    # Log detailed response and reasoning if available
    if hasattr(run, 'metadata'):
        logger.info(f"Run metadata: {run.metadata}")
    else:
        logger.warning("Metadata not found in the run object.")

# Save logs to a file
logging.info("Run completed with status: %s", run.status)
logging.info("Run details: %s", run.to_json(indent=2))

# If run is incomplete, log the reason
if run.status == "incomplete" and hasattr(run, 'incomplete_details'):
    logging.info("Run incomplete, reason: %s", run.incomplete_details.get('reason', 'No reason provided'))

# If the run is completed and has metadata, log it
if run.status == "completed" and hasattr(run, 'metadata'):
    logging.info("Run metadata: %s", run.metadata)

. Hope this helps.

Mayank Goel 0 Reputation points

2024-12-10T18:13:04.57+00:00
Thanks for your quick response. I tried all of this but this is what I am seeing:

metadata parameter in both run/message object is always empty and is never getting populated.

I was expecting below reasoning/plan/action but not seeing it anywhere:
As an example, lets say, for timesheet submission user needs to fill some mandate fields then plan that LLM comes might look like:

To submit your timesheet, I need:

Dates worked (e.g., "2024-12-01 to 2024-12-02").

Hours worked (e.g., "8, 8").

Task description (e.g., "Worked on project X").
I am looking to get this information of plan that everytime gets updated and gets stored ib thread however metadata field of messages is always coming as {} blank

Also, for my cases, run.status is coming as completed so I am good there and don't have to worry about reason but as mentioned above my main intention is to get the above kind of plan which is always missing in metadata and could see that even one you have shared is blank. Any here here will be highly appreciated. Thanks
navba-MSFT 27,485 Reputation points Microsoft Employee

2024-12-11T14:07:11.7966667+00:00
@Mayank Goel Thanks for getting back. Even in the documentations I see the metadata is blank {}.

There are alternative ways to achieve this without relying on the metadata field. Here are a few approaches you can consider:

.

Custom Logging within the Assistant's Responses:

You can modify the assistant's responses to include the plan and reasoning directly within the message content. This way, you can parse the message content to extract the necessary information.

.

Structured Responses:

Design the assistant's responses to follow a structured format that includes the plan and reasoning. For example, you can use specific keywords or delimiters to separate different parts of the response.

.

Intermediate Steps Logging:

Log each step of the assistant's reasoning and planning process as separate messages within the thread. This way, you can track the progression of the assistant's thought process.

. Here's an example of how you might implement structured responses:

import logging # Function to create a structured response def create_structured_response(dates, hours, task_description): plan = f"Plan: To submit your timesheet, I need:\n- Dates worked: {dates}\n- Hours worked: {hours}\n- Task description: {task_description}" reasoning = "Reasoning: These fields are required to accurately record your work hours and tasks." return f"{plan}\n\n{reasoning}" # Example usage dates = "2024-12-01 to 2024-12-02" hours = "8, 8" task_description = "Worked on project X" response = create_structured_response(dates, hours, task_description) logging.info(response)

In this example, the create_structured_response function generates a response that includes both the plan and reasoning. You can then log or process this response as needed.

Use of Tags or Annotations:

You can use tags or annotations within the message content to mark different sections. For example, you can use [PLAN] and [REASONING] tags to identify the respective parts of the response.

response = """ [PLAN] To submit your timesheet, I need: - Dates worked: 2024-12-01 to 2024-12-02 - Hours worked: 8, 8 - Task description: Worked on project X [REASONING] These fields are required to accurately record your work hours and tasks. """ # Log the response logging.info(response)

By using these approaches, you can ensure that the plan and reasoning are included in the assistant's responses and can be easily extracted and logged without relying on the metadata field.

Your answer

navba-MSFT 27,485 Reputation points Microsoft Employee

2024-12-09T04:24:10.7433333+00:00

@Mayank Goel Just following up to check if the below answer helped. If that answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let me know. I would be happy to help.
Mayank Goel 0 Reputation points

2024-12-09T21:53:57.5733333+00:00

I tried both the approaches and I am attaching run object here.Sample_Run.txt From here, we can see that run object doesn't have response/reasoning hence it is never giving any action/plan and below logging code doesn't yield anything:

if 'response' in run and 'reasoning' in run['response']:

logger.info(f"Assistant reasoning: {run['response']['reasoning']}") ```All I can see is there is response_format='auto' however no parameter related to plan/action/reasoning in run object at least. Even the message object doesn't have any action/plan/reasoning object/parameter. Can you please look into this and help?
Mayank Goel 0 Reputation points

2024-12-10T18:13:04.57+00:00

Thanks for your quick response. I tried all of this but this is what I am seeing:

metadata parameter in both run/message object is always empty and is never getting populated.

I was expecting below reasoning/plan/action but not seeing it anywhere:
As an example, lets say, for timesheet submission user needs to fill some mandate fields then plan that LLM comes might look like:

To submit your timesheet, I need:

Dates worked (e.g., "2024-12-01 to 2024-12-02").

Hours worked (e.g., "8, 8").

Task description (e.g., "Worked on project X").
I am looking to get this information of plan that everytime gets updated and gets stored ib thread however metadata field of messages is always coming as {} blank

Also, for my cases, run.status is coming as completed so I am good there and don't have to worry about reason but as mentioned above my main intention is to get the above kind of plan which is always missing in metadata and could see that even one you have shared is blank. Any here here will be highly appreciated. Thanks

Share via

How to Retrieve Action/Plan/Reasoning with OpenaAI Assistant

Your answer