Edit

How to use tool calling with Foundry Local

Foundry Local can make use of tool calling, a technique where you prompt the model with definitions of available tools that together with a text prompt, allow the model to work out which tools should be called and with what input data. The application then calls those tools and adds the results to a subsequent model prompt to answer the user's query.

The tools can perform functions that the model doesn't have access to, such as getting the current weather, or reading files on the local file system, or accessing a user's address book (providing the application has permission to do so).

This guide shows you how to use this feature of Foundry Local.

Models that support tool calling

Using the Foundry Local CLI, you can run the foundry model list command to see which models support tool calling.

In the Task column, you can see that the tools task indicates that tool calling is supported.

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/tool-calling-foundry-local-sdk

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

dotnet add package Microsoft.AI.Foundry.Local.WinML
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Understanding tool choice settings

The tool choice parameter controls whether and how the model invokes the tools you provide. Tool choice is sent as part of the chat completion request alongside your tool definitions.

Different models have different capabilities when it comes to tool calling, but in general you can expect the following behavior for each option:

Option Value Behavior Reliability
Auto "auto" The model decides whether to call a tool or respond directly, based on the user's message and the available tool definitions. Reliable across all tool-calling models
None "none" The model won't call any tools, even if tools are provided in the request. Reliable across all tool-calling models
Required "required" The model must call at least one tool. Best-effort
Specific function {"type": "function", "function": {"name": "my_function"}} The model must call the specified function. Best-effort

Use native chat completions with tool calling

Copy and paste the following code into a C# file named Program.cs:

using Microsoft.AI.Foundry.Local;
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Betalgo.Ranul.OpenAI.ObjectModels.ResponseModels;
using Betalgo.Ranul.OpenAI.ObjectModels.SharedModels;
using System.Text.Json;

CancellationToken ct = new CancellationToken();

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};


// Initialize the singleton instance.
await FoundryLocalManager.CreateAsync(config, Utils.GetAppLogger());
var mgr = FoundryLocalManager.Instance;


// Ensure that any Execution Provider (EP) downloads run and are completed.
// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
    if (epName != currentEp)
    {
        if (currentEp != "") Console.WriteLine();
        currentEp = epName;
    }
    Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();


// Get the model catalog
var catalog = await mgr.GetCatalogAsync();


// Get a model using an alias.
var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");


// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});


// Load the model
Console.Write($"Loading model {model.Id}...");
await model.LoadAsync();
Console.WriteLine("done.");


// Get a chat client
var chatClient = await model.GetChatClientAsync();
chatClient.Settings.ToolChoice = ToolChoice.Required; // Force the model to make a tool call


// Prepare messages
List<ChatMessage> messages =
[
    new ChatMessage { Role = "system", Content = "You are a helpful AI assistant. If necessary, you can use any provided tools to answer the question." },
    new ChatMessage { Role = "user", Content = "What is the answer to 7 multiplied by 6?" }
];


// Prepare tools
List<ToolDefinition> tools =
[
    new ToolDefinition
    {
        Type = "function",
        Function = new FunctionDefinition()
        {
            Name = "multiply_numbers",
            Description = "A tool for multiplying two numbers.",
            Parameters = new PropertyDefinition()
            {
                Type = "object",
                Properties = new Dictionary<string, PropertyDefinition>()
                {
                    { "first", new PropertyDefinition() { Type = "integer", Description = "The first number in the operation" } },
                    { "second", new PropertyDefinition() { Type = "integer", Description = "The second number in the operation" } }
                },
                Required = ["first", "second"]
            }
        }
    }
];


// Get a streaming chat completion response
var toolCallResponses = new List<ChatCompletionCreateResponse>();
Console.WriteLine("Chat completion response:");
var streamingResponse = chatClient.CompleteChatStreamingAsync(messages, tools, ct);
await foreach (var chunk in streamingResponse)
{
    var content = chunk.Choices[0].Message.Content;
    Console.Write(content);
    Console.Out.Flush();

    if (chunk.Choices[0].FinishReason == "tool_calls")
    {
        toolCallResponses.Add(chunk);
    }
}
Console.WriteLine();


// Invoke tools called and append responses to the chat
foreach (var chunk in toolCallResponses)
{
    var call = chunk?.Choices[0].Message.ToolCalls?[0].FunctionCall;
    if (call?.Name == "multiply_numbers")
    {
        var arguments = JsonSerializer.Deserialize<Dictionary<string, int>>(call.Arguments!)!;
        var first = arguments["first"];
        var second = arguments["second"];

        Console.WriteLine($"\nInvoking tool: {call?.Name} with arguments {first} and {second}");
        var result = Utils.MultiplyNumbers(first, second);
        Console.WriteLine($"Tool response: {result.ToString()}");

        var response = new ChatMessage
        {
            Role = "tool",
            ToolCallId = chunk!.Choices[0].Message.ToolCalls![0].Id,
            Content = result.ToString(),
        };
        messages.Add(response);
    }
}
Console.WriteLine("\nTool calls completed. Prompting model to continue conversation...\n");


// Prompt the model to continue the conversation after the tool call
messages.Add(new ChatMessage { Role = "system", Content = "Respond only with the answer generated by the tool." });


// Set tool calling back to auto so that the model can decide whether to call
// the tool again or continue the conversation based on the new user prompt
chatClient.Settings.ToolChoice = ToolChoice.Auto;


// Run the next turn of the conversation
Console.WriteLine("Chat completion response:");
streamingResponse = chatClient.CompleteChatStreamingAsync(messages, tools, ct);
await foreach (var chunk in streamingResponse)
{
    var content = chunk.Choices[0].Message.Content;
    Console.Write(content);
    Console.Out.Flush();
}
Console.WriteLine();


// Tidy up - unload the model
await model.UnloadAsync();

Run the native chat completions example

dotnet run

Use OpenAI Web server for tool calling

If you prefer to use the OpenAI SDKs to call the Foundry Local web service, use the following example that demonstrates how to handle tool calling in that scenario.

Tip

Use options.ToolChoice = ChatToolChoice.CreateAutoChoice(); (the default) for the most reliable behavior. Write clear tool names and descriptions so the model calls the correct tool on its own.

using Microsoft.AI.Foundry.Local;
using OpenAI;
using OpenAI.Chat;
using System.ClientModel;
using System.Text.Json;

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information,
    Web = new Configuration.WebService
    {
        Urls = "http://127.0.0.1:52495"
    }
};


// Initialize the singleton instance.
await FoundryLocalManager.CreateAsync(config, Utils.GetAppLogger());
var mgr = FoundryLocalManager.Instance;


// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
    if (epName != currentEp)
    {
        if (currentEp != "") Console.WriteLine();
        currentEp = epName;
    }
    Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();


// Get the model catalog
var catalog = await mgr.GetCatalogAsync();


// Get a model using an alias
var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");
// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});


// Load the model
Console.Write($"Loading model {model.Id}...");
await model.LoadAsync();
Console.WriteLine("done.");


// Start the web service
Console.Write($"Starting web service on {config.Web.Urls}...");
await mgr.StartWebServiceAsync();
Console.WriteLine("done.");


// <<<<<< OPEN AI SDK USAGE >>>>>>
// Use the OpenAI SDK to call the local Foundry web service

ApiKeyCredential key = new ApiKeyCredential("notneeded");
OpenAIClient client = new OpenAIClient(key, new OpenAIClientOptions
{
    Endpoint = new Uri(config.Web.Urls + "/v1"),
});


// Get chat client
var chatClient = client.GetChatClient(model.Id);


// Prepare messages
var messages = new List<ChatMessage>
{
    ChatMessage.CreateSystemMessage("You are a helpful AI assistant. If necessary, you can use any provided tools to answer the question."),
    ChatMessage.CreateUserMessage("What is the answer to 7 multiplied by 6?")
};


// Prepare tools
var tools = new List<ChatTool>
{
    ChatTool.CreateFunctionTool(
        functionName: "multiply_numbers",
        functionDescription: "A tool for multiplying two numbers.",
        functionParameters: BinaryData.FromString("""
        {
            "type": "object",
            "properties": {
                "first": { "type": "number", "description": "The first number in the operation" },
                "second": { "type": "number", "description": "The second number in the operation" }
            },
            "required": ["first", "second"]
        }
        """)
    )
};


// Prepare chat completion options
var options = new ChatCompletionOptions
{
    ToolChoice = ChatToolChoice.CreateRequiredChoice()  // Force the model to make a tool call
};
foreach (var tool in tools)
{
    options.Tools.Add(tool);
}


// Get a streaming chat completion response
var completionUpdates = chatClient.CompleteChatStreaming(messages, options);
var toolCalls = new List<StreamingChatToolCallUpdate>();
Console.Write($"[ASSISTANT]: ");
foreach (var completionUpdate in completionUpdates)
{
    if (completionUpdate.ContentUpdate.Count > 0)
    {
        Console.Write(completionUpdate.ContentUpdate[0].Text);
    }

    if (completionUpdate.FinishReason == ChatFinishReason.ToolCalls)
    {
        foreach (var toolCall in completionUpdate.ToolCallUpdates)
        {
            toolCalls.Add(toolCall);
        }
    }
}
Console.WriteLine();


// Invoke tools called and append responses to the chat
foreach (var toolCall in toolCalls)
{
    if (toolCall.FunctionName == "multiply_numbers")
    {
        var arguments = JsonDocument.Parse(toolCall.FunctionArgumentsUpdate.ToString()).RootElement;
        var first = arguments.GetProperty("first").GetInt32();
        var second = arguments.GetProperty("second").GetInt32();

        Console.WriteLine($"\nInvoking tool: {toolCall.FunctionName} with arguments {first} and {second}");
        var result = Utils.MultiplyNumbers(first, second);
        Console.WriteLine($"Tool response: {result.ToString()}");

        messages.Add(ChatMessage.CreateToolMessage(toolCallId: "abcd1234", content: result.ToString()));
    }
}
Console.WriteLine("\nTool calls completed. Prompting model to continue conversation...\n");


// Prompt the model to continue the conversation after the tool call
messages.Add(ChatMessage.CreateSystemMessage("Respond only with the answer generated by the tool."));


// Set tool calling back to auto so that the model can decide whether to call
// the tool again or continue the conversation based on the new user prompt
options.ToolChoice = ChatToolChoice.CreateAutoChoice();


// Run the next turn of the conversation
Console.WriteLine("Chat completion response:");
completionUpdates = chatClient.CompleteChatStreaming(messages, options);
Console.Write($"[ASSISTANT]: ");
foreach (var completionUpdate in completionUpdates)
{
    if (completionUpdate.ContentUpdate.Count > 0)
    {
        Console.Write(completionUpdate.ContentUpdate[0].Text);
    }
}
Console.WriteLine();

// <<<<<< END OPEN AI SDK USAGE >>>>>>


// Tidy up
// Stop the web service and unload model
await mgr.StopWebServiceAsync();
await model.UnloadAsync();

Run the OpenAI web service example

dotnet run

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/tool-calling-foundry-local

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

npm install foundry-local-sdk-winml openai

Understanding tool choice settings

The tool choice parameter controls whether and how the model invokes the tools you provide. Tool choice is sent as part of the chat completion request alongside your tool definitions.

Different models have different capabilities when it comes to tool calling, but in general you can expect the following behavior for each option:

Option Value Behavior Reliability
Auto "auto" The model decides whether to call a tool or respond directly, based on the user's message and the available tool definitions. Reliable across all tool-calling models
None "none" The model doesn't invoke any tools, even if tools are provided in the request. Reliable across all tool-calling models
Required "required" The model must call at least one tool. Best-effort
Specific function {"type": "function", "function": {"name": "my_function"}} The model must call the specified function. Best-effort

Use chat completions with tool calling

Copy and paste the following code into a JavaScript file named app.js:

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

import { OpenAI } from "openai";
import { FoundryLocalManager } from "foundry-local-sdk";

// By using an alias, the most suitable model will be downloaded 
// to your end-user's device.
// TIP: You can find a list of available models by running the 
// following command in your terminal: `foundry model list`.
const alias = "qwen2.5-0.5b";

function multiplyNumbers(first, second) {
  return first * second;
}

async function runToolCallingExample() {
  let manager = null;
  let model = null;

  try {
    console.log("Initializing Foundry Local SDK...");
    manager = FoundryLocalManager.create({
      appName: "foundry_local_samples",
      serviceEndpoint: "http://localhost:5000",
      logLevel: "info"
    });

    // Download and register all execution providers.
    let currentEp = '';
    await manager.downloadAndRegisterEps((epName, percent) => {
      if (epName !== currentEp) {
        if (currentEp !== '') process.stdout.write('\n');
        currentEp = epName;
      }
      process.stdout.write(`\r  ${epName.padEnd(30)}  ${percent.toFixed(1).padStart(5)}%`);
    });
    if (currentEp !== '') process.stdout.write('\n');

    const catalog = manager.catalog;
    model = await catalog.getModel(alias);
    if (!model) {
      throw new Error(`Model ${alias} not found`);
    }

    console.log(`Loading model ${model.id}...`);
    await model.download();
    await model.load();
    console.log('✓ Model loaded');

    manager.startWebService();
    const endpoint = manager.urls[0];
    if (!endpoint) {
      throw new Error("Foundry Local web service did not return an endpoint.");
    }

    const openai = new OpenAI({
      baseURL: `${endpoint.replace(/\/$/, "")}/v1`,
      apiKey: "local"
    });

    // Prepare messages
    const messages = [
      {
        role: "system",
        content: "You are a helpful AI assistant. If necessary, you can use any provided tools to answer the question."
      },
      { role: "user", content: "What is the answer to 7 multiplied by 6?" }
    ];

    // Prepare tools
    const tools = [
      {
        type: "function",
        function: {
          name: "multiply_numbers",
          description: "A tool for multiplying two numbers.",
          parameters: {
            type: "object",
            properties: {
              first: {
                type: "integer",
                description: "The first number in the operation"
              },
              second: {
                type: "integer",
                description: "The second number in the operation"
              }
            },
            required: ["first", "second"]
          }
        }
      }
    ];

    // Start the conversation
    console.log("Chat completion response:");
    const toolCallResponses = [];

    const firstStream = await openai.chat.completions.create({
      model: model.id,
      messages,
      tools,
      tool_choice: "required",
      stream: true
    });

    for await (const chunk of firstStream) {
      const content = chunk.choices?.[0]?.delta?.content;
      if (content) {
        process.stdout.write(content);
      }

      if (chunk.choices?.[0]?.finish_reason === "tool_calls") {
        toolCallResponses.push(chunk);
      }
    }
    console.log();

    // Invoke tools called and append responses to the chat
    for (const chunk of toolCallResponses) {
      const toolCalls = chunk.choices?.[0]?.message?.tool_calls ?? chunk.choices?.[0]?.delta?.tool_calls ?? [];
      for (const toolCall of toolCalls) {
        if (toolCall.function?.name === "multiply_numbers") {
          const args = JSON.parse(toolCall.function.arguments || "{}");
          const first = args.first;
          const second = args.second;

          console.log(`\nInvoking tool: ${toolCall.function.name} with arguments ${first} and ${second}`);
          const result = multiplyNumbers(first, second);
          console.log(`Tool response: ${result}`);

          messages.push({
            role: "tool",
            tool_call_id: toolCall.id,
            content: result.toString()
          });
        }
      }
    }

    console.log("\nTool calls completed. Prompting model to continue conversation...\n");

    // Prompt the model to continue the conversation after the tool call
    messages.push({
      role: "system",
      content: "Respond only with the answer generated by the tool."
    });

    // Run the next turn of the conversation
    console.log("Chat completion response:");
    const secondStream = await openai.chat.completions.create({
      model: model.id,
      messages,
      tools,
      tool_choice: "auto",
      stream: true
    });

    for await (const chunk of secondStream) {
      const content = chunk.choices?.[0]?.delta?.content;
      if (content) {
        process.stdout.write(content);
      }
    }

    console.log();
  } finally {
    if (model) {
      try {
        if (await model.isLoaded()) {
          await model.unload();
        }
      } catch (cleanupError) {
        console.warn("Cleanup warning while unloading model:", cleanupError);
      }
    }

    if (manager) {
      try {
        manager.stopWebService();
      } catch (cleanupError) {
        console.warn("Cleanup warning while stopping service:", cleanupError);
      }
    }
  }
}

await runToolCallingExample().catch((error) => {
  console.error("Error running sample:", error);
  process.exitCode = 1;
});

To run the application, execute the following command in your terminal:

node app.js

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/tool-calling

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

pip install foundry-local-sdk-winml openai

Install the OpenAI SDK:

pip install openai

Understanding tool choice settings

The tool choice parameter controls whether and how the model invokes the tools you provide. The parameter is sent as part of the chat completion request alongside your tool definitions.

Different models have different capabilities when it comes to tool calling, but in general you can expect the following behavior for each option:

Option Value Behavior Reliability
Auto "auto" The model decides whether to call a tool or respond directly, based on the user's message and the available tool definitions. Reliable across all tool-calling models
None "none" The model won't call any tools, even if tools are provided in the request. Reliable across all tool-calling models
Required "required" The model must call at least one tool. Best-effort
Specific function {"type": "function", "function": {"name": "my_function"}} The model must call the specified function. Best-effort

Use chat completions with tool calling

Copy and paste the following code into a Python file named app.py:

import json
from foundry_local_sdk import Configuration, FoundryLocalManager



# --- Tool definitions ---
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city or location",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit",
                    },
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "calculate",
            "description": "Perform a math calculation",
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": ("The math expression to evaluate"),
                    }
                },
                "required": ["expression"],
            },
        },
    },
]


# --- Tool implementations ---
def get_weather(location, unit="celsius"):
    """Simulate a weather lookup."""
    return {
        "location": location,
        "temperature": 22 if unit == "celsius" else 72,
        "unit": unit,
        "condition": "Sunny",
    }


def calculate(expression):
    """Evaluate a math expression safely."""
    allowed = set("0123456789+-*/(). ")
    if not all(c in allowed for c in expression):
        return {"error": "Invalid expression"}
    try:
        result = eval(expression)
        return {"expression": expression, "result": result}
    except Exception as e:
        return {"error": str(e)}


tool_functions = {"get_weather": get_weather, "calculate": calculate}


def process_tool_calls(messages, response, client):
    """Handle tool calls in a loop until the model produces a final answer."""
    choice = response.choices[0].message

    while choice.tool_calls:
        # Convert the assistant message to a dict for the SDK
        assistant_msg = {
            "role": "assistant",
            "content": choice.content,
            "tool_calls": [
                {
                    "id": tc.id,
                    "type": tc.type,
                    "function": {
                        "name": tc.function.name,
                        "arguments": tc.function.arguments,
                    },
                }
                for tc in choice.tool_calls
            ],
        }
        messages.append(assistant_msg)

        for tool_call in choice.tool_calls:
            function_name = tool_call.function.name
            arguments = json.loads(tool_call.function.arguments)
            print(f"  Tool call: {function_name}({arguments})")

            # Execute the function and add the result
            func = tool_functions[function_name]
            result = func(**arguments)
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "content": json.dumps(result),
                }
            )

        # Send the updated conversation back
        response = client.complete_chat(messages, tools=tools)
        choice = response.choices[0].message

    return choice.content




def main():
    # Initialize the Foundry Local SDK
    config = Configuration(app_name="foundry_local_samples")
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    # Download and register all execution providers.
    current_ep = ""

    def ep_progress(ep_name: str, percent: float):
        nonlocal current_ep
        if ep_name != current_ep:
            if current_ep:
                print()
            current_ep = ep_name
        print(f"\r  {ep_name:<30}  {percent:5.1f}%", end="", flush=True)

    manager.download_and_register_eps(progress_callback=ep_progress)
    if current_ep:
        print()

    # Select and load a model
    model = manager.catalog.get_model("qwen2.5-0.5b")
    model.download(
        lambda progress: print(
            f"\rDownloading model: {progress:.2f}%", end="", flush=True
        )
    )
    print()
    model.load()
    print("Model loaded and ready.")

    # Get a chat client
    client = model.get_chat_client()

    # Conversation with a system prompt
    messages = [
        {
            "role": "system",
            "content": "You are a helpful assistant with access to tools. "
            "Use them when needed to answer questions accurately.",
        },
        {
            "role": "user",
            "content": "What is the weather in Seattle and what is 42 * 17?",
        },
    ]

    print("Sending request with tools...")
    response = client.complete_chat(messages, tools=tools)
    answer = process_tool_calls(messages, response, client)

    print(f"\nAssistant: {answer}")

    # Clean up
    model.unload()
    print("Model unloaded.")




if __name__ == "__main__":
    main()

To run the application, execute the following command in your terminal:

python app.py

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/tool-calling-foundry-local

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

cargo add foundry-local-sdk --features winml
cargo add tokio --features full
cargo add tokio-stream anyhow

Add extra dependencies for HTTP and JSON:

cargo add anyhow reqwest --features reqwest/json
cargo add serde serde_json --features serde/derive

Understanding tool choice settings

The tool choice parameter controls whether and how the model invokes the tools you provide. Tool choice is sent as part of the chat completion request alongside your tool definitions.

Different models have different capabilities when it comes to tool calling, but in general you can expect the following behavior for each option:

Option Value Behavior Reliability
Auto "auto" The model decides whether to call a tool or respond directly, based on the user's message and the available tool definitions. Reliable across all tool-calling models
None "none" The model doesn't call any tools, even if tools are provided in the request. Reliable across all tool-calling models
Required "required" The model must call at least one tool. Best effort (tool call could be ignored by smaller models)
Specific function {"type": "function", "function": {"name": "my_function"}} The model must call the specified function. Best-effort (tool call could be ignored by smaller models)

Use chat completions with tool calling

Replace the contents of src/main.rs with the following code:

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

use std::io::{self, Write};

use serde_json::{json, Value};
use tokio_stream::StreamExt;

use foundry_local_sdk::{
    ChatCompletionRequestMessage, ChatCompletionRequestSystemMessage,
    ChatCompletionRequestToolMessage, ChatCompletionRequestUserMessage, ChatCompletionTools,
    ChatToolChoice, FinishReason, FoundryLocalConfig, FoundryLocalManager,
};

// By using an alias, the most suitable model variant will be downloaded
// to your end-user's device.
const ALIAS: &str = "qwen2.5-0.5b";

/// A simple tool that multiplies two numbers.
fn multiply_numbers(first: f64, second: f64) -> f64 {
    first * second
}

/// Dispatch a tool call by name and parsed arguments.
fn invoke_tool(name: &str, args: &Value) -> String {
    match name {
        "multiply_numbers" => {
            let first = args.get("first").and_then(|v| v.as_f64()).unwrap_or(0.0);
            let second = args.get("second").and_then(|v| v.as_f64()).unwrap_or(0.0);
            let result = multiply_numbers(first, second);
            result.to_string()
        }
        _ => format!("Unknown tool: {name}"),
    }
}

/// Accumulated state from a streaming response that contains tool calls.
#[derive(Default)]
struct ToolCallState {
    tool_calls: Vec<Value>,
    current_tool_id: String,
    current_tool_name: String,
    current_tool_args: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("Tool Calling with Foundry Local");
    println!("===============================\n");

    // ── 1. Initialise the manager ────────────────────────────────────────
    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;

    // Download and register all execution providers.
    manager
        .download_and_register_eps_with_progress(None, {
            let mut current_ep = String::new();
            move |ep_name: &str, percent: f64| {
                if ep_name != current_ep {
                    if !current_ep.is_empty() {
                        println!();
                    }
                    current_ep = ep_name.to_string();
                }
                print!("\r  {:<30}  {:5.1}%", ep_name, percent);
                io::stdout().flush().ok();
            }
        })
        .await?;
    println!();

    // ── 2. Load a model──────────────────────────────────────────────────
    let model = manager.catalog().get_model(ALIAS).await?;
    println!("Model: {} (id: {})", model.alias(), model.id());

    if !model.is_cached().await? {
        println!("Downloading model...");
        model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    println!("Loading model...");
    model.load().await?;
    println!("✓ Model loaded\n");

    // ── 3. Create a chat clientwith tool_choice = required ──────────────
    let client = model.create_chat_client()
        .max_tokens(512)
        .tool_choice(ChatToolChoice::Required);

    // Define the multiply_numbers tool.
    let tools: Vec<ChatCompletionTools> = serde_json::from_value(json!([{
        "type": "function",
        "function": {
            "name": "multiply_numbers",
            "description": "A tool for multiplying two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "first": {
                        "type": "integer",
                        "description": "The first number in the operation"
                    },
                    "second": {
                        "type": "integer",
                        "description": "The second number in the operation"
                    }
                },
                "required": ["first", "second"]
            }
        }
    }]))?;

    // Prepare the initial conversation.
    let mut messages: Vec<ChatCompletionRequestMessage> = vec![
        ChatCompletionRequestSystemMessage::from(
            "You are a helpful AI assistant. If necessary, you can use any provided tools to answer the question.",
        )
        .into(),
        ChatCompletionRequestUserMessage::from("What is the answer to 7 multiplied by 6?").into(),
    ];

    // ── 4. First streaming call – expect tool_calls ──────────────────────
    println!("Chat completion response:");

    let mut state = ToolCallState::default();
    let mut stream = client
        .complete_streaming_chat(&messages, Some(&tools))
        .await?;

    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        if let Some(choice) = chunk.choices.first() {
            // Accumulate streamed content (if any).
            if let Some(ref content) = choice.delta.content {
                print!("{content}");
                io::stdout().flush().ok();
            }

            // Accumulate tool call fragments.
            if let Some(ref tool_calls) = choice.delta.tool_calls {
                for tc in tool_calls {
                    if let Some(ref id) = tc.id {
                        state.current_tool_id = id.clone();
                    }
                    if let Some(ref func) = tc.function {
                        if let Some(ref name) = func.name {
                            state.current_tool_name = name.clone();
                        }
                        if let Some(ref args) = func.arguments {
                            state.current_tool_args.push_str(args);
                        }
                    }
                }
            }

            // When the model signals finish_reason = ToolCalls, finalise.
            if choice.finish_reason == Some(FinishReason::ToolCalls) {
                let tc = json!({
                    "id": state.current_tool_id.clone(),
                    "type": "function",
                    "function": {
                        "name": state.current_tool_name.clone(),
                        "arguments": state.current_tool_args.clone(),
                    }
                });
                state.tool_calls.push(tc);
            }
        }
    }
    println!();

    // ── 5. Execute the tool(s)and append results ────────────────────────
    for tc in &state.tool_calls {
        let func = &tc["function"];
        let name = func["name"].as_str().unwrap_or_default();
        let args_str = func["arguments"].as_str().unwrap_or("{}");
        let args: Value = serde_json::from_str(args_str).unwrap_or(json!({}));

        println!("\nInvoking tool: {name} with arguments {args}");
        let result = invoke_tool(name, &args);
        println!("Tool response: {result}");

        // Append the assistant's tool_calls message and the tool result.
        let assistant_msg: ChatCompletionRequestMessage = serde_json::from_value(json!({
            "role": "assistant",
            "content": null,
            "tool_calls": [tc],
        }))?;
        messages.push(assistant_msg);
        messages.push(
            ChatCompletionRequestToolMessage {
                content: result.into(),
                tool_call_id: tc["id"].as_str().unwrap_or_default().to_string(),
            }
            .into(),
        );
    }

    // ── 6. Continue the conversation with auto tool_choice ───────────────
    println!("\nTool calls completed. Prompting model to continue conversation...\n");

    messages.push(
        ChatCompletionRequestSystemMessage::from(
            "Respond only with the answer generated by the tool.",
        )
        .into(),
    );

    let client = client.tool_choice(ChatToolChoice::Auto);

    print!("Chat completion response: ");
    let mut stream = client
        .complete_streaming_chat(&messages, Some(&tools))
        .await?;
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        if let Some(choice) = chunk.choices.first() {
            if let Some(ref content) = choice.delta.content {
                print!("{content}");
                io::stdout().flush().ok();
            }
        }
    }
    println!("\n");

    // ── 7. Clean up──────────────────────────────────────────────────────
    println!("Unloading model...");
    model.unload().await?;
    println!("Done.");

    Ok(())
}

To run the application, execute the following command in your terminal:

cargo run

Related content

  • Last updated on