Tutorial: Build a voice-to-text note taker

Build an application that converts spoken audio into organized notes — entirely on your device. The app first transcribes an audio file using a speech-to-text model, then uses a chat model to summarize and organize the transcription into clean notes.

In this tutorial, you learn how to:

• Set up a project and install the Foundry Local SDK
• Load a speech-to-text model and transcribe an audio file
• Load a chat model and summarize the transcription
• Combine transcription and summarization into a complete app
• Clean up resources

Prerequisites

• A Windows, macOS, or Linux computer with at least 8 GB of RAM.
• A .wav audio file to transcribe (the tutorial uses a sample file).

Install packages

Samples repository

You can find the complete sample code for this article in the Foundry samples GitHub repository. To clone the repository and navigate to the sample, use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/tutorial-voice-to-text

dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Transcribe an audio file

In this step, you load a speech-to-text model and transcribe an audio file. The Foundry Local SDK uses the whisper model alias to select the best Whisper variant for your hardware.

• Open Program.cs and replace its contents with the following code to initialize the SDK, load the speech model, and transcribe an audio file:

  // Load the speech-to-text model
  var speechModel = await catalog.GetModelAsync("whisper-tiny")
      ?? throw new Exception("Speech model not found");
  
  await speechModel.DownloadAsync(progress =>
  {
      Console.Write($"\rDownloading speech model: {progress:F2}%");
      if (progress >= 100f) Console.WriteLine();
  });
  
  await speechModel.LoadAsync();
  Console.WriteLine("Speech model loaded.");
  
  // Transcribe the audio file with the native v2 audio session.
  var transcriptionText = new StringBuilder();
  using var audioSession = new AudioSession(speechModel);
  audioSession.SetStreaming(true);
  using var audioRequest = new Request();
  audioRequest.AddItem(new AudioItem("meeting-notes.wav"));
  
  Console.WriteLine("\nTranscription:");
  await using var audioResponse = audioSession.ProcessStreamingRequestAsync(audioRequest, ct);
  await foreach (var item in audioResponse)
  {
      using (item)
      {
          if (item is SpeechSegmentItem segment)
          {
              Console.Write(segment.Text);
              transcriptionText.Append(segment.Text);
          }
      }
  }
  using var finalTranscription = await audioResponse.FinalResponse;
  Console.WriteLine();
  
  // Unload the speech model to free memory
  finalTranscription.Dispose();
  await audioResponse.DisposeAsync();
  audioRequest.Dispose();
  audioSession.Dispose();
  await speechModel.UnloadAsync();
  

  The GetAudioClientAsync method returns a client for audio operations. The TranscribeAudioStreamingAsync method streams transcription chunks as they become available. You accumulate the text so you can pass it to the chat model in the next step.

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Summarize the transcription

Now use a chat model to organize the raw transcription into structured notes. Load the qwen2.5-0.5b model and send the transcription as context with a system prompt that instructs the model to produce clean, summarized notes.

Add the following code after the transcription step:

// Load the chat model for summarization
var chatModel = await catalog.GetModelAsync("qwen2.5-0.5b")
    ?? throw new Exception("Chat model not found");

await chatModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading chat model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await chatModel.LoadAsync();
Console.WriteLine("Chat model loaded.");

// Summarize the transcription into organized notes.
using var chatSession = new ChatSession(chatModel);
using var chatRequest = new Request();
chatRequest.AddItem(MessageItem.System(
    "You are a note-taking assistant. Summarize the following transcription " +
    "into organized, concise notes with bullet points."));
chatRequest.AddItem(MessageItem.User(transcriptionText.ToString()));
using var chatResponse = await chatSession.ProcessRequestAsync(chatRequest, ct);
var summary = string.Concat(
    chatResponse.OfType<MessageItem>()
        .SelectMany(message => message.Parts)
        .OfType<TextItem>()
        .Where(text => text.Type == TextItemType.Default)
        .Select(text => text.Text));
Console.WriteLine($"\nSummary:\n{summary}");

// Clean up
chatResponse.Dispose();
chatRequest.Dispose();
chatSession.Dispose();
await chatModel.UnloadAsync();
Console.WriteLine("\nDone. Models unloaded.");

The system prompt shapes the model's output format. By instructing it to produce "organized, concise notes with bullet points," you get structured content rather than a raw paraphrase.

Combine into a complete app

Replace the contents of Program.cs with the following complete code that transcribes an audio file and summarizes the transcription:

using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging;
using System.Text;

CancellationToken ct = CancellationToken.None;

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.SetMinimumLevel(
        Microsoft.Extensions.Logging.LogLevel.Information
    );
});
var logger = loggerFactory.CreateLogger<Program>();

// Initialize the singleton instance
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
    if (epName != currentEp)
    {
        if (currentEp != "") Console.WriteLine();
        currentEp = epName;
    }
    Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();

var catalog = await mgr.GetCatalogAsync();

// Load the speech-to-text model
var speechModel = await catalog.GetModelAsync("whisper-tiny")
    ?? throw new Exception("Speech model not found");

await speechModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading speech model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await speechModel.LoadAsync();
Console.WriteLine("Speech model loaded.");

// Transcribe the audio file with the native v2 audio session.
var transcriptionText = new StringBuilder();
using var audioSession = new AudioSession(speechModel);
audioSession.SetStreaming(true);
using var audioRequest = new Request();
audioRequest.AddItem(new AudioItem("meeting-notes.wav"));

Console.WriteLine("\nTranscription:");
await using var audioResponse = audioSession.ProcessStreamingRequestAsync(audioRequest, ct);
await foreach (var item in audioResponse)
{
    using (item)
    {
        if (item is SpeechSegmentItem segment)
        {
            Console.Write(segment.Text);
            transcriptionText.Append(segment.Text);
        }
    }
}
using var finalTranscription = await audioResponse.FinalResponse;
Console.WriteLine();

// Unload the speech model to free memory
finalTranscription.Dispose();
await audioResponse.DisposeAsync();
audioRequest.Dispose();
audioSession.Dispose();
await speechModel.UnloadAsync();

// Load the chat model for summarization
var chatModel = await catalog.GetModelAsync("qwen2.5-0.5b")
    ?? throw new Exception("Chat model not found");

await chatModel.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading chat model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await chatModel.LoadAsync();
Console.WriteLine("Chat model loaded.");

// Summarize the transcription into organized notes.
using var chatSession = new ChatSession(chatModel);
using var chatRequest = new Request();
chatRequest.AddItem(MessageItem.System(
    "You are a note-taking assistant. Summarize the following transcription " +
    "into organized, concise notes with bullet points."));
chatRequest.AddItem(MessageItem.User(transcriptionText.ToString()));
using var chatResponse = await chatSession.ProcessRequestAsync(chatRequest, ct);
var summary = string.Concat(
    chatResponse.OfType<MessageItem>()
        .SelectMany(message => message.Parts)
        .OfType<TextItem>()
        .Where(text => text.Type == TextItemType.Default)
        .Select(text => text.Text));
Console.WriteLine($"\nSummary:\n{summary}");

// Clean up
chatResponse.Dispose();
chatRequest.Dispose();
chatSession.Dispose();
await chatModel.UnloadAsync();
Console.WriteLine("\nDone. Models unloaded.");

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Run the note taker:

dotnet run

You see output similar to:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

The application first transcribes the audio content with streaming output, then passes the accumulated text to a chat model that extracts key points and organizes them into structured notes.

Install packages

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/tutorial-voice-to-text

npm install foundry-local-sdk openai

Transcribe an audio file

In this step, you load a speech-to-text model and transcribe an audio file. The Foundry Local SDK uses the whisper model alias to select the best Whisper variant for your hardware.

1. Create a file called app.js.

2. Add the following code to initialize the SDK, load the speech model, and transcribe an audio file:

   // Load the speech-to-text model
   const speechModel = await manager.catalog.getModel('whisper-tiny');
   await speechModel.download((progress) => {
       process.stdout.write(
           `\rDownloading speech model: ${progress.toFixed(2)}%`
       );
   });
   console.log('\nSpeech model downloaded.');
   
   await speechModel.load();
   console.log('Speech model loaded.');
   
   // Transcribe the audio file
   const audioSession = new AudioSession(speechModel);
   const audioFile = path.join(__dirname, 'meeting-notes.wav');
   const transcriptionResponse = await audioSession.processRequest(
       new Request().addItem(Item.audioFromUri(audioFile))
   );
   const transcription = transcriptionResponse.output.find(
       (item) => item.type === 'speechResult'
   );
   if (!transcription) {
       throw new Error('The speech model returned no transcription.');
   }
   console.log(`\nTranscription:\n${transcription.text}`);
   
   // Unload the speech model to free memory
   audioSession.dispose();
   await speechModel.unload();
   

   The createAudioClient method returns a client for audio operations. The transcribe method accepts a file path and returns an object with a text property containing the transcribed content.

Note

Replace './meeting-notes.wav' with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Summarize the transcription

Now use a chat model to organize the raw transcription into structured notes. Load the qwen2.5-0.5b model and send the transcription as context with a system prompt that instructs the model to produce clean, summarized notes.

Add the following code after the transcription step:

// Load the chat model for summarization
const chatModel = await manager.catalog.getModel('qwen2.5-0.5b');
await chatModel.download((progress) => {
    process.stdout.write(
        `\rDownloading chat model: ${progress.toFixed(2)}%`
    );
});
console.log('\nChat model downloaded.');

await chatModel.load();
console.log('Chat model loaded.');

// Summarize the transcription into organized notes
const chatSession = new ChatSession(chatModel);
const request = new Request()
    .addItem(Item.systemMessage(
        'You are a note-taking assistant. Summarize ' +
        'the following transcription into organized, ' +
        'concise notes with bullet points.'
    ))
    .addItem(Item.userMessage(transcription.text));

const response = await chatSession.processRequest(request);
const summary = response.output
    .map((item) => item.type === 'message'
        ? item.content ?? item.parts?.map((part) => part.text ?? '').join('') ?? ''
        : item.type === 'text' ? item.text : '')
    .join('');
console.log(`\nSummary:\n${summary}`);

// Clean up
chatSession.dispose();
await chatModel.unload();
console.log('\nDone. Models unloaded.');

The system prompt shapes the model's output format. By instructing it to produce "organized, concise notes with bullet points," you get structured content rather than a raw paraphrase.

Combine into a complete app

Create a file named app.js and add the following complete code that transcribes an audio file and summarizes the transcription:

import {
    AudioSession,
    ChatSession,
    FoundryLocalManager,
    Item,
    Request
} from 'foundry-local-sdk';
import { fileURLToPath } from 'url';
import path from 'path';

const __dirname = path.dirname(fileURLToPath(import.meta.url));

// Initialize the Foundry Local SDK
const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});

// Download and register all execution providers.
let currentEp = '';
await manager.downloadAndRegisterEps((epName, percent) => {
    if (epName !== currentEp) {
        if (currentEp !== '') process.stdout.write('\n');
        currentEp = epName;
    }
    process.stdout.write(`\r  ${epName.padEnd(30)}  ${percent.toFixed(1).padStart(5)}%`);
});
if (currentEp !== '') process.stdout.write('\n');

// Load the speech-to-text model
const speechModel = await manager.catalog.getModel('whisper-tiny');
await speechModel.download((progress) => {
    process.stdout.write(
        `\rDownloading speech model: ${progress.toFixed(2)}%`
    );
});
console.log('\nSpeech model downloaded.');

await speechModel.load();
console.log('Speech model loaded.');

// Transcribe the audio file
const audioSession = new AudioSession(speechModel);
const audioFile = path.join(__dirname, 'meeting-notes.wav');
const transcriptionResponse = await audioSession.processRequest(
    new Request().addItem(Item.audioFromUri(audioFile))
);
const transcription = transcriptionResponse.output.find(
    (item) => item.type === 'speechResult'
);
if (!transcription) {
    throw new Error('The speech model returned no transcription.');
}
console.log(`\nTranscription:\n${transcription.text}`);

// Unload the speech model to free memory
audioSession.dispose();
await speechModel.unload();

// Load the chat model for summarization
const chatModel = await manager.catalog.getModel('qwen2.5-0.5b');
await chatModel.download((progress) => {
    process.stdout.write(
        `\rDownloading chat model: ${progress.toFixed(2)}%`
    );
});
console.log('\nChat model downloaded.');

await chatModel.load();
console.log('Chat model loaded.');

// Summarize the transcription into organized notes
const chatSession = new ChatSession(chatModel);
const request = new Request()
    .addItem(Item.systemMessage(
        'You are a note-taking assistant. Summarize ' +
        'the following transcription into organized, ' +
        'concise notes with bullet points.'
    ))
    .addItem(Item.userMessage(transcription.text));

const response = await chatSession.processRequest(request);
const summary = response.output
    .map((item) => item.type === 'message'
        ? item.content ?? item.parts?.map((part) => part.text ?? '').join('') ?? ''
        : item.type === 'text' ? item.text : '')
    .join('');
console.log(`\nSummary:\n${summary}`);

// Clean up
chatSession.dispose();
await chatModel.unload();
console.log('\nDone. Models unloaded.');

Note

Replace './meeting-notes.wav' with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Run the note taker:

node app.js

You see output similar to:

Downloading speech model: 100.00%
Speech model downloaded.
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model downloaded.
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

The application first transcribes the audio content, then passes that text to a chat model that extracts key points and organizes them into structured notes.

Install packages

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/tutorial-voice-to-text

pip install foundry-local-sdk openai

Transcribe an audio file

In this step, you load a speech-to-text model and transcribe an audio file. The Foundry Local SDK uses the whisper model alias to select the best Whisper variant for your hardware.

1. Create a file called app.py.

2. Add the following code to initialize the SDK, load the speech model, and transcribe an audio file:

   # Load the speech-to-text model.
   speech_model = manager.catalog.get_model("whisper-tiny")
   if speech_model is None:
       raise RuntimeError("Model 'whisper-tiny' was not found in the catalog.")
   download_if_needed(speech_model, "speech")
   speech_model.load()
   print("Speech model loaded.")
   
   try:
       # Transcribe the audio file with a native audio session.
       with AudioSession(speech_model) as session:
           with Request().add_item(AudioItem.from_uri(audio_path)) as request:
               with session.process_request(request) as response:
                   transcription = get_text(response)
       print(f"\nTranscription:\n{transcription}")
   finally:
       # Unload the speech model to free memory.
       if speech_model.is_loaded:
           speech_model.unload()
   

   The get_audio_client method returns a client for audio operations. The transcribe method accepts a file path and returns an object with a text property containing the transcribed content.

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Summarize the transcription

Now use a chat model to organize the raw transcription into structured notes. Load the qwen2.5-0.5b model and send the transcription as context with a system prompt that instructs the model to produce clean, summarized notes.

Add the following code after the transcription step:

# Load the chat model for summarization.
chat_model = manager.catalog.get_model("qwen2.5-0.5b")
if chat_model is None:
    manager.close()
    raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
download_if_needed(chat_model, "chat")
chat_model.load()
print("Chat model loaded.")

try:
    # Summarize the transcription into organized notes.
    with ChatSession(chat_model) as session:
        session.set_options(
            RequestOptions(
                search=SearchOptions(temperature=0.2, max_output_tokens=512)
            )
        )
        with (
            Request()
            .add_item(
                MessageItem.system(
                    "You are a note-taking assistant. Summarize the following "
                    "transcription into organized, concise notes with bullet points."
                )
            )
            .add_item(MessageItem.user(transcription)) as request
        ):
            with session.process_request(request) as response:
                summary = get_text(response)
    print(f"\nSummary:\n{summary}")
finally:
    # Clean up.
    if chat_model.is_loaded:
        chat_model.unload()
    manager.close()
    print("\nDone. Models unloaded.")

The system prompt shapes the model's output format. By instructing it to produce "organized, concise notes with bullet points," you get structured content rather than a raw paraphrase.

Combine into a complete app

Create a file named app.py and add the following complete code that transcribes an audio file and summarizes the transcription:

import sys

from foundry_local_sdk import (
    AudioItem,
    AudioSession,
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    SpeechResultItem,
    TextItem,
)



def download_if_needed(model, label):
    if model.is_cached:
        print(f"Using cached {label} model {model.id}.")
        return
    model.download(
        lambda progress: print(
            f"\rDownloading {label} model: {progress:.2f}%",
            end="",
            flush=True,
        )
    )
    print()


def get_text(response):
    parts = []
    for item in response:
        if isinstance(item, SpeechResultItem):
            parts.append(item.text)
        elif isinstance(item, TextItem):
            parts.append(item.text)
        elif isinstance(item, MessageItem):
            parts.extend(part.text for part in item.parts if isinstance(part, TextItem))
    return "".join(parts)


def main():
    # Initialize the Foundry Local SDK
    config = Configuration(app_name="foundry_local_samples")
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    audio_path = sys.argv[1] if len(sys.argv) > 1 else "meeting-notes.wav"

    # Load the speech-to-text model.
    speech_model = manager.catalog.get_model("whisper-tiny")
    if speech_model is None:
        raise RuntimeError("Model 'whisper-tiny' was not found in the catalog.")
    download_if_needed(speech_model, "speech")
    speech_model.load()
    print("Speech model loaded.")

    try:
        # Transcribe the audio file with a native audio session.
        with AudioSession(speech_model) as session:
            with Request().add_item(AudioItem.from_uri(audio_path)) as request:
                with session.process_request(request) as response:
                    transcription = get_text(response)
        print(f"\nTranscription:\n{transcription}")
    finally:
        # Unload the speech model to free memory.
        if speech_model.is_loaded:
            speech_model.unload()

    # Load the chat model for summarization.
    chat_model = manager.catalog.get_model("qwen2.5-0.5b")
    if chat_model is None:
        manager.close()
        raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
    download_if_needed(chat_model, "chat")
    chat_model.load()
    print("Chat model loaded.")

    try:
        # Summarize the transcription into organized notes.
        with ChatSession(chat_model) as session:
            session.set_options(
                RequestOptions(
                    search=SearchOptions(temperature=0.2, max_output_tokens=512)
                )
            )
            with (
                Request()
                .add_item(
                    MessageItem.system(
                        "You are a note-taking assistant. Summarize the following "
                        "transcription into organized, concise notes with bullet points."
                    )
                )
                .add_item(MessageItem.user(transcription)) as request
            ):
                with session.process_request(request) as response:
                    summary = get_text(response)
        print(f"\nSummary:\n{summary}")
    finally:
        # Clean up.
        if chat_model.is_loaded:
            chat_model.unload()
        manager.close()
        print("\nDone. Models unloaded.")


if __name__ == "__main__":
    main()

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Run the note taker:

python app.py

You see output similar to:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

The application first transcribes the audio content, then passes that text to a chat model that extracts key points and organizes them into structured notes.

Install packages

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/tutorial-voice-to-text

cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow

Transcribe an audio file

In this step, you load a speech-to-text model and transcribe an audio file. The Foundry Local SDK uses the whisper model alias to select the best Whisper variant for your hardware.

• Open src/main.rs and replace its contents with the following code to initialize the SDK, load the speech model, and transcribe an audio file:

  let speech_model = manager.catalog().get_model("whisper-tiny").await?;
  
  if !speech_model.is_cached().await? {
      println!("Downloading speech model...");
      speech_model
          .download(Some(|progress: f64| {
              print!("\r  {progress:.1}%");
              io::stdout().flush().ok();
          }))
          .await?;
      println!();
  }
  
  speech_model.load().await?;
  println!("Speech model loaded.");
  
  let audio_session = AudioSession::new(&speech_model).await?;
  let audio_path = std::fs::canonicalize("meeting-notes.wav")?;
  let audio_uri = Url::from_file_path(&audio_path)
      .map_err(|_| format!("Could not convert {} to a file URI.", audio_path.display()))?;
  let audio = Item::audio_uri(audio_uri.as_str(), Some("wav"), 0, 0);
  let transcription = audio_session.transcribe(audio).await?;
  println!("\nTranscription:\n{transcription}");
  
  drop(audio_session);
  speech_model.unload().await?;
  

  The create_audio_client method returns a client for audio operations. The transcribe method accepts a file path and returns an object with a text field containing the transcribed content.

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Summarize the transcription

Now use a chat model to organize the raw transcription into structured notes. Load the qwen2.5-0.5b model and send the transcription as context with a system prompt that instructs the model to produce clean, summarized notes.

Add the following code after the transcription step, inside the main function:

let chat_model = manager.catalog().get_model("qwen2.5-0.5b").await?;

if !chat_model.is_cached().await? {
    println!("Downloading chat model...");
    chat_model
        .download(Some(|progress: f64| {
            print!("\r  {progress:.1}%");
            io::stdout().flush().ok();
        }))
        .await?;
    println!();
}

chat_model.load().await?;
println!("Chat model loaded.");

let chat_session = ChatSession::new(&chat_model).await?;
chat_session
    .set_options(RequestOptions {
        search: SearchOptions {
            temperature: Some(0.7),
            max_output_tokens: Some(512),
            ..Default::default()
        },
        ..Default::default()
    })
    .await?;

let response = chat_session
    .process_request(Request::from_items(vec![
        Item::system_message(vec![Item::text(
            "You are a note-taking assistant. Summarize the following transcription into \
             organized, concise notes with bullet points.",
        )]),
        Item::user_message(vec![Item::text(transcription)]),
    ]))
    .await?;

println!("\nSummary:");
for item in &response.items {
    if let Some(message) = item.as_message() {
        print!("{}", message.text());
    } else if let Some(text) = item.as_text() {
        print!("{text}");
    }
}
println!();

drop(chat_session);
chat_model.unload().await?;
println!("\nDone. Models unloaded.");

The system prompt shapes the model's output format. By instructing it to produce "organized, concise notes with bullet points," you get structured content rather than a raw paraphrase.

Combine into a complete app

Replace the contents of src/main.rs with the following complete code that transcribes an audio file and summarizes the transcription:

use foundry_local_sdk::{
    AudioSession, ChatSession, FoundryLocalConfig, FoundryLocalManager, Item, Request,
    RequestOptions, SearchOptions,
};
use std::io::{self, Write};
use url::Url;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize the Foundry Local SDK
    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;

    manager
        .download_and_register_eps_with_progress(None, {
            let mut current_ep = String::new();
            move |ep_name: &str, percent: f64| {
                if ep_name != current_ep {
                    if !current_ep.is_empty() {
                        println!();
                    }
                    current_ep = ep_name.to_string();
                }
                print!("\r  {ep_name:<30}  {percent:5.1}%");
                io::stdout().flush().ok();
            }
        })
        .await?;
    println!();

    let speech_model = manager.catalog().get_model("whisper-tiny").await?;

    if !speech_model.is_cached().await? {
        println!("Downloading speech model...");
        speech_model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    speech_model.load().await?;
    println!("Speech model loaded.");

    let audio_session = AudioSession::new(&speech_model).await?;
    let audio_path = std::fs::canonicalize("meeting-notes.wav")?;
    let audio_uri = Url::from_file_path(&audio_path)
        .map_err(|_| format!("Could not convert {} to a file URI.", audio_path.display()))?;
    let audio = Item::audio_uri(audio_uri.as_str(), Some("wav"), 0, 0);
    let transcription = audio_session.transcribe(audio).await?;
    println!("\nTranscription:\n{transcription}");

    drop(audio_session);
    speech_model.unload().await?;

    let chat_model = manager.catalog().get_model("qwen2.5-0.5b").await?;

    if !chat_model.is_cached().await? {
        println!("Downloading chat model...");
        chat_model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    chat_model.load().await?;
    println!("Chat model loaded.");

    let chat_session = ChatSession::new(&chat_model).await?;
    chat_session
        .set_options(RequestOptions {
            search: SearchOptions {
                temperature: Some(0.7),
                max_output_tokens: Some(512),
                ..Default::default()
            },
            ..Default::default()
        })
        .await?;

    let response = chat_session
        .process_request(Request::from_items(vec![
            Item::system_message(vec![Item::text(
                "You are a note-taking assistant. Summarize the following transcription into \
                 organized, concise notes with bullet points.",
            )]),
            Item::user_message(vec![Item::text(transcription)]),
        ]))
        .await?;

    println!("\nSummary:");
    for item in &response.items {
        if let Some(message) = item.as_message() {
            print!("{}", message.text());
        } else if let Some(text) = item.as_text() {
            print!("{text}");
        }
    }
    println!();

    drop(chat_session);
    chat_model.unload().await?;
    println!("\nDone. Models unloaded.");

    Ok(())
}

Note

Replace "meeting-notes.wav" with the path to your audio file. Supported formats include WAV, MP3, and FLAC.

Run the note taker:

cargo run

You see output similar to:

Downloading speech model: 100.00%
Speech model loaded.

Transcription:
OK so let's get started with the weekly sync. First, the backend
API is nearly done. Sarah finished the authentication endpoints
yesterday. We still need to add rate limiting before we go to
staging. On the frontend, the dashboard redesign is about seventy
percent complete. Jake, can you walk us through the new layout?
Great. The charts look good. I think we should add a filter for
date range though. For testing, we have about eighty percent code
coverage on the API. We need to write integration tests for the
new auth flow before Friday. Let's plan to do a full regression
test next Tuesday before the release. Any blockers? OK, sounds
like we are in good shape. Let's wrap up.

Downloading chat model: 100.00%
Chat model loaded.

Summary:
- **Backend API**: Authentication endpoints complete. Rate limiting
  still needed before staging deployment.
- **Frontend**: Dashboard redesign 70% complete. New chart layout
  reviewed. Action item: add a date range filter.
- **Testing**: API code coverage at 80%. Integration tests for the
  auth flow due Friday. Full regression test scheduled for next
  Tuesday before release.
- **Status**: No blockers reported. Team is on track.

Done. Models unloaded.

The application first transcribes the audio content, then passes that text to a chat model that extracts key points and organizes them into structured notes.

Samples repository

Clone the Foundry samples GitHub repository and go to the C++ sample:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/cpp/foundry-local/tutorial-voice-to-text

Install packages

The C++ samples require:

• CMake 3.20 or later.
• A C++20 compiler.
• The NuGet CLI on your PATH.

Configure and build the sample:

cmake -S . -B build
cmake --build build --config Release

During configuration, the sample restores version 2.2.1 of the Microsoft.AI.Foundry.Local.Runtime NuGet package to build/_packages. The package contains the C++ headers and native libraries for Windows, Linux, and macOS. You don't need a separate WinML package.

The CMake configuration selects the runtime assets for your operating system and architecture and copies the required native libraries beside the executable. On macOS and Linux, run the executable from build. On Windows, run the .exe file from build.

Transcribe an audio file

Load a speech model and transcribe an audio file:

auto speech_model = manager.GetCatalog().GetModel("whisper-tiny");
if (!speech_model) {
  throw std::runtime_error("Model 'whisper-tiny' was not found in the catalog.");
}
DownloadIfNeeded(*speech_model, "speech");
speech_model->Load();

std::string transcript;
{
  AudioSession session(*speech_model);
  Request request{Item::AudioFromUri(argv[1])};
  transcript = ResponseText(session.ProcessRequest(request));
}
speech_model->Unload();
std::cout << "Transcription:\n" << transcript << "\n";

Summarize the transcription

Load a chat model and turn the transcription into structured notes:

auto chat_model = manager.GetCatalog().GetModel("qwen2.5-0.5b");
if (!chat_model) {
  throw std::runtime_error("Model 'qwen2.5-0.5b' was not found in the catalog.");
}
DownloadIfNeeded(*chat_model, "chat");
chat_model->Load();

{
  ChatSession session(*chat_model);
  Request request{
      SystemMessage(
          "Turn the transcription into concise meeting notes with action items."),
      UserMessage(transcript),
  };
  request.SetOptions({
      .search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
  });
  std::cout << "Summary:\n" << ResponseText(session.ProcessRequest(request)) << "\n";
}
chat_model->Unload();

Combine the steps

#include <foundry_local/foundry_local_cpp.h>

#include <iostream>
#include <string>

using namespace foundry_local;

void DownloadIfNeeded(IModel& model, const std::string& label) {
  if (model.IsCached()) {
    std::cout << "Using cached " << label << " model.\n";
    return;
  }
  model.Download([&label](float progress) {
    std::cout << "\rDownloading " << label << " model: " << static_cast<int>(progress) << "%"
              << std::flush;
    return 0;
  });
  std::cout << "\n";
}

std::string ResponseText(const Response& response) {
  std::string text;
  for (const Item& item : response.GetItems()) {
    if (item.GetType() == FOUNDRY_LOCAL_ITEM_SPEECH_RESULT) {
      text.append(item.GetSpeechResult().text);
    } else if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT) {
      text.append(item.GetText().text);
    } else if (item.GetType() == FOUNDRY_LOCAL_ITEM_MESSAGE) {
      for (const Item& part : item.GetMessage().parts) {
        if (part.GetType() == FOUNDRY_LOCAL_ITEM_TEXT) {
          text.append(part.GetText().text);
        }
      }
    }
  }
  return text;
}

int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Usage: tutorial-voice-to-text <audio-file>\n";
    return 1;
  }

  try {
    Configuration config("foundry_local_cpp_voice_to_text");
    Manager manager(std::move(config));

    auto speech_model = manager.GetCatalog().GetModel("whisper-tiny");
    if (!speech_model) {
      throw std::runtime_error("Model 'whisper-tiny' was not found in the catalog.");
    }
    DownloadIfNeeded(*speech_model, "speech");
    speech_model->Load();

    std::string transcript;
    {
      AudioSession session(*speech_model);
      Request request{Item::AudioFromUri(argv[1])};
      transcript = ResponseText(session.ProcessRequest(request));
    }
    speech_model->Unload();
    std::cout << "Transcription:\n" << transcript << "\n";

    auto chat_model = manager.GetCatalog().GetModel("qwen2.5-0.5b");
    if (!chat_model) {
      throw std::runtime_error("Model 'qwen2.5-0.5b' was not found in the catalog.");
    }
    DownloadIfNeeded(*chat_model, "chat");
    chat_model->Load();

    {
      ChatSession session(*chat_model);
      Request request{
          SystemMessage(
              "Turn the transcription into concise meeting notes with action items."),
          UserMessage(transcript),
      };
      request.SetOptions({
          .search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
      });
      std::cout << "Summary:\n" << ResponseText(session.ProcessRequest(request)) << "\n";
    }
    chat_model->Unload();

    manager.Shutdown();
  } catch (const Error& error) {
    std::cerr << "Foundry Local error [" << error.Code() << "]: " << error.what() << "\n";
    return 1;
  } catch (const std::exception& error) {
    std::cerr << "Error: " << error.what() << "\n";
    return 1;
  }

  return 0;
}

Run the note taker with an audio file path:

./build/tutorial-voice-to-text path/to/audio.wav

Clean up resources

The model weights remain in your local cache after you unload a model. This means the next time you run the application, the download step is skipped and the model loads faster. No extra cleanup is needed unless you want to reclaim disk space.

Related content

• Get started with Foundry Local
• Transcribe audio (speech-to-text)
• Tutorial: Build a multi-turn chat assistant