Fine-tune a model with reinforcement fine-tuning in Microsoft Foundry

Prepare your data and grader, then choose a response format and training settings. Create a reinforcement fine-tuning (RFT) job in Microsoft Foundry. For method comparisons and training concepts, see the fine-tuning overview.

Prerequisites

• A Foundry resource with an RFT-supported model and training region, including any required model access approval.
• The Foundry User role for training, or the Foundry Owner role if you also deploy a fine-tuned model or grader model.

• The client packages and credentials described in the Python procedure.

• Your resource endpoint and credentials, and a Bash-compatible shell for the REST examples.

• Azure Developer CLI version 1.22.1 or later.

Important

The Foundry RBAC roles were recently renamed. Foundry User, Foundry Owner, Foundry Account Owner, and Foundry Project Manager were previously named Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. You might still see the previous names in some places while the rename rolls out. The role IDs and core permissions are unchanged by the rename.

Prepare your data

Start with the MedMCQ sample datasets on GitHub, or prepare your own data.

Prepare separate training and validation files in the format supported by your selected model. Each example includes a prompt and the reference information your grader needs. Keep validation and final test examples out of the training set.

Save the datasets as training.jsonl and validation.jsonl, with one example per line. Each record has a messages array ending in a user message, plus any fields your grader uses. The MedMCQ sample uses reference_answer for the expected answer label.

Reference: RFT dataset samples.

Use the dataset format and file paths in your selected RFT CLI configuration sample.

Understand how graders work

A grader is the reward function for your RFT job. The model generates responses, the grader scores them, and training uses those scores to update the model.

Define the behavior you want to reward before choosing a grader. Use a reliable reference answer, an objective check, or a clear scoring rubric.

Choose a grader type

Choose a grading approach supported by your selected model:

Grader type	When to use it
String comparison	Exact answers, labels, or substring checks
Text similarity	Responses that should resemble a known reference
Model grader	Responses that require judgment against a rubric
Python grader	Task-specific calculations or custom checks
Multi-Grader	Tasks with more than one scoring criterion

Match each example's references to your dataset and response format.

String comparison

Compare the response with an expected answer or label. For example, award 1 for an exact match and 0 for any other response.

This definition uses the MedMCQ sample's expected answer label:

{
  "name": "medmcqa_ans_grader",
  "type": "string_check",
  "input": "{{item.reference_answer}}",
  "reference": "{{sample.output_text}}",
  "operation": "eq"
}

Reference: String-check grader schema and MedMCQ grader.

Parameter	Type	Required	Description
type	String	Yes	Set to string_check
name	String	Yes	A name for the grader
input	String	Yes	The text to test; can contain template references
reference	String	Yes	The text to compare against; can contain template references
operation	String	Yes	The comparison operation

Choose the operation that matches your task:

Operation	Returns 1 when	Case-sensitive
eq	Input equals reference	Yes
ne	Input differs from reference	Yes
like	Input contains reference	Yes
ilike	Input contains reference	No

For a containment check, put the generated response in input and the expected substring in reference.

Text similarity

Compare the response with a reference using a text-similarity metric. For example, grade an extracted passage against the expected passage instead of requiring an exact match.

This definition uses the extracted-text check from the ClauseMatching sample. It requires structured output with an extracted_text field:

{
  "name": "extracted_text_similarity",
  "type": "text_similarity",
  "input": "{{sample.output_json.extracted_text}}",
  "reference": "{{item.extracted_text}}",
  "evaluation_metric": "fuzzy_match"
}

Reference: Text-similarity grader schema and ClauseMatching grader.

Parameter	Type	Required	Description
type	String	Yes	Set to text_similarity
name	String	Yes	A name for the grader
input	String	Yes	The generated text to grade; can contain template references
reference	String	Yes	The expected text; can contain template references
evaluation_metric	String	Yes	The supported similarity metric

Choose a metric supported by your RFT workflow:

Metric	Comparison
fuzzy_match	Approximate string matching
bleu	BLEU n-gram overlap
gleu	Google BLEU
meteor	METEOR text alignment
rouge_1 through rouge_5	N-gram overlap, from unigrams through five-grams
rouge_l	Longest common subsequence

Use cosine for evaluations, not RFT.

Model grader

Give a grading model a rubric for judging the response. For example, award 1 for a correct answer, 0.5 for a partially correct answer, and 0 for an incorrect answer.

Replace <SUPPORTED_GRADER_MODEL_ID> with a model supported for grading in your RFT workflow. Check the grader deployment requirements before creating the job.

This example adapts the documented score-model rubric to a dataset with reference_answer:

{
  "name": "answer_quality",
  "type": "score_model",
  "model": "<SUPPORTED_GRADER_MODEL_ID>",
  "input": [
    {
      "role": "system",
      "content": "Score 1 for a match, 0.5 for a partial match, or 0."
    },
    {
      "role": "user",
      "content": "Reference: {{item.reference_answer}}"
    },
    {
      "role": "user",
      "content": "Response: {{sample.output_text}}"
    }
  ],
  "range": [0, 1],
  "sampling_params": {
    "seed": 42
  }
}

Reference: Score-model grader schema.

Parameter	Type	Required	Description
type	String	Yes	Set to score_model
name	String	Yes	A name for the grader
model	String	Yes	The supported grading-model identifier, not the model you're training
input	Array of messages	Yes	The rubric and response to grade; messages include a role and content, which can contain template references
range	Array of numbers	No	The lower and upper score bounds; this example uses [0, 1]
sampling_params	Object	No	Sampling settings for the grading model, separate from training hyperparameters

Use only sampling settings supported by the grading model:

Parameter in sampling_params	Type	Description
seed	Integer or null	A sampling seed; don't assume it guarantees identical scores
temperature	Number or null	Controls sampling randomness, when supported
top_p	Number or null	Controls nucleus sampling
max_completions_tokens	Integer or null	Limits tokens generated by the grading model
reasoning_effort	String	Controls reasoning effort for a grading model that supports it; accepted values depend on that model

The example's seed is illustrative, not a recommended default. Test the rubric on responses with known scores before training.

Python grader

Use Python code to calculate a numeric reward from the response and reference data. For example, compare the answer with its reference or implement a task-specific calculation.

Define a grade(sample, item) function that returns a numeric score. This helper saves an exact-answer grader as grader.json:

import json

source = """
def grade(sample, item):
    return float(sample["output_text"] == item["reference_answer"])
"""
grader = {
    "name": "answer_match",
    "type": "python",
    "source": source,
}
with open("grader.json", "w", encoding="utf-8") as grader_file:
    json.dump(grader, grader_file)

Reference: Python grader schema.

The helper encodes the function as the JSON source string, including its newline characters. It defines the grader, not the training job; use your selected interface to submit the job.

Parameter	Type	Required	Description
type	String	Yes	Set to python
name	String	Yes	A name for the grader
source	String	Yes	Python source defining grade(sample, item); the function returns a numeric score
image_tag	String	No	A runtime image tag supported by your workflow; omit it unless you need a documented image version

The function receives the generated response in sample and the dataset record in item. It returns 1.0 for a match and 0.0 otherwise. Ensure every record supplies reference_answer.

Python graders run in a constrained runtime without network access. Keep code within the documented runtime limits, and test error cases before training.

Multi-Grader

Combine multiple grading criteria into one reward. For example, give equal weight to the quality of an extracted passage and the correctness of its classification.

This definition combines the two checks in the ClauseMatching sample. Use the matching dataset and structured response format:

{
  "name": "clause_match_grader",
  "type": "multi",
  "graders": {
    "extracted_text_similarity": {
      "name": "extracted_text_similarity",
      "type": "text_similarity",
      "input": "{{sample.output_json.extracted_text}}",
      "reference": "{{item.extracted_text}}",
      "evaluation_metric": "fuzzy_match"
    },
    "clause_string_check": {
      "name": "clause_string_check",
      "type": "string_check",
      "input": "{{sample.output_json.clause_type}}",
      "reference": "{{item.clause_type}}",
      "operation": "eq"
    }
  },
  "calculate_output": "(extracted_text_similarity + clause_string_check) / 2"
}

Reference: Multi-Grader schema and ClauseMatching Multi-Grader.

Parameter	Type	Required	Description
type	String	Yes	Set to multi
name	String	Yes	A name for the combined grader
graders	Object	Yes	Named child-grader definitions, each using a supported grader type and its required parameters
calculate_output	String	Yes	An arithmetic expression that references child graders by their keys in graders

The expression supports +, -, *, /, and ^, plus min, max, abs, floor, ceil, exp, sqrt, and log. Choose child-grader combinations supported by your RFT workflow.

Reference model output and dataset fields

Align your grader with the response you expect and the reference information in your dataset. A grader that expects a plain answer label doesn't correctly evaluate a structured response containing that label.

Templated graders substitute these references when scoring a response:

Reference	Content
{{sample.output_text}}	The generated response as text
{{sample.output_json}}	The generated structured response as JSON
{{item.reference_answer}}	The expected answer in the current dataset record
{{item.extracted_text}} and {{item.clause_type}}	Reference fields in the ClauseMatching dataset

Every reference must match an actual response or dataset field. Python graders receive the same information through the sample and item function arguments instead of template substitution.

Model grader deployments

A model grader uses a grading model, which can differ from the model you're fine-tuning. Depending on the selected training model, Foundry hosts the grader or requires a separate grader deployment.

If your workflow requires a separate deployment, provision a supported grader model before training. Confirm access, capacity, and the grading budget.

Configure and test the grader

Test the grader on correct, incorrect, and malformed responses. Correct answers should receive higher scores than incorrect answers. Check that superficial responses can't earn a high score without solving the task.

Configure the grader in the fine-tuning job setup using the options available for your selected model. For an exact-answer task, use a string comparison that rewards only the expected answer.

For the exact-answer sample, save the string-comparison definition as grader.json. Configure your prompts to request only the answer label.

For other tasks, use a matching configuration from the grader samples.

After initializing the client and loading the grader in the SDK example below, validate it with client.fine_tuning.alpha.graders.validate(grader=grader). Test sample responses with client.fine_tuning.alpha.graders.run, providing the grader, generated response, and dataset record.

Reference: Grader validation and test APIs.

Use the grader validation and test APIs before submitting the job.

Choose a grader configuration sample for your task. Update the grader and dataset paths in the YAML configuration, and check that its scoring rules match your dataset.

Choose a response format

Choose an output format supported by your model and compatible with the grader. Use plain text for an answer-label task, or structured JSON when your task requires named output fields.

Set the response format in the job configuration. For structured output, provide the matching schema and use a grader that evaluates that structure.

For plain text, leave response_format unset and grade {{sample.output_text}}. For structured output, supply a schema in method.reinforcement.response_format and grade {{sample.output_json}} or its fields.

The following configuration wraps the ClauseMatching sample schema. Save it as response-format.json only when using the matching dataset and grader.

{
  "type": "json_schema",
  "json_schema": {
    "name": "contract_clause",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "clause_type": {
          "type": "string",
          "enum": ["Exclusivity", "Non-Compete"]
        },
        "extracted_text": {
          "type": "string"
        }
      },
      "required": ["clause_type", "extracted_text"],
      "additionalProperties": false
    }
  }
}

Reference: RFT response-format schema and structured outputs.

Align the prompts, dataset fields, and grader with that output. Don't reuse an exact-text grader unchanged after switching to JSON.

Keep the response format consistent with the grader in your YAML configuration. Use the matching dataset and grader files from the RFT CLI samples.

Configure hyperparameters

Start with your selected model's defaults. Compare validation rewards and grader behavior before adjusting training, sampling, or evaluation settings.

Use the settings available in the job configuration. The available controls and accepted values depend on your selected model.

The v1 examples below omit optional hyperparameters. To override defaults, use supported settings under method.reinforcement.hyperparameters in the v1 fine-tuning API.

Keep the default settings in your model's YAML configuration for your first job. Use the configuration sample for your selected model rather than copying settings from another model.

Create an RFT job in the portal

Use your prepared datasets and grader to create the job:

1. Open Build > Fine-tune in the Foundry portal, and select Fine-tune.
2. Select an eligible model and the reinforcement fine-tuning method.
3. Upload the training and validation datasets.
4. Configure and test the grader. For a model grader, check who hosts the grader deployment.
5. Choose a response format supported by your model, and use a matching grader.
6. Select a supported training type, and leave hyperparameters at their defaults for your first job.
7. Review the configuration and create the job. Retain its ID.

Create an RFT job with Python

Initialize the client with the Microsoft Foundry SDK for your project, or use the OpenAI SDK with a resource endpoint. Both clients use the same upload and job commands below.

Foundry SDK

Install azure-ai-projects, azure-identity, and openai. Sign in with a credential supported by DefaultAzureCredential, and set FOUNDRY_PROJECT_ENDPOINT to your project endpoint:

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
)
client = project.get_openai_client()

Reference: AIProjectClient and DefaultAzureCredential.

OpenAI SDK

Install openai and set AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY for your resource:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["AZURE_OPENAI_ENDPOINT"].rstrip("/") + "/openai/v1/",
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
)

Reference: Azure OpenAI client configuration.

Set FINE_TUNING_MODEL to an identifier from supported model IDs and versions that supports RFT with this API.

Upload the files, load and validate grader.json, and submit the job with default hyperparameters. For structured output, also set RFT_RESPONSE_FORMAT_PATH to your response-format JSON file and use a matching grader and dataset.

import json
import os

with open("training.jsonl", "rb") as training:
    training_file = client.files.create(file=training, purpose="fine-tune")
with open("validation.jsonl", "rb") as validation:
    validation_file = client.files.create(
        file=validation, purpose="fine-tune"
    )
client.files.wait_for_processing(training_file.id)
client.files.wait_for_processing(validation_file.id)
with open("grader.json", encoding="utf-8") as grader_file:
    grader = json.load(grader_file)
client.fine_tuning.alpha.graders.validate(grader=grader)

reinforcement = {
    "grader": grader,
}
response_format_path = os.environ.get("RFT_RESPONSE_FORMAT_PATH")
if response_format_path:
    with open(response_format_path, encoding="utf-8") as format_file:
        reinforcement["response_format"] = json.load(format_file)

job = client.fine_tuning.jobs.create(
    model=os.environ["FINE_TUNING_MODEL"],
    training_file=training_file.id,
    validation_file=validation_file.id,
    extra_body={
        "method": {
            "type": "reinforcement",
            "reinforcement": reinforcement,
        },
    },
    suffix="my-rft-model",
)
print(job.id, job.status)

Reference: Fine-tuning API.

The example sends the RFT method configuration in extra_body and prints the job ID and status. Leave RFT_RESPONSE_FORMAT_PATH unset for the plain-text example.

Create an RFT job with REST

Set AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY for your resource. Choose a model from supported model IDs and versions that supports RFT with this API.

In a Bash-compatible shell, upload both datasets and retain their distinct file IDs:

curl -X POST "$AZURE_OPENAI_ENDPOINT/openai/v1/files" \
  -H "api-key: $AZURE_OPENAI_API_KEY" \
  -F "purpose=fine-tune" -F "file=@training.jsonl"

curl -X POST "$AZURE_OPENAI_ENDPOINT/openai/v1/files" \
  -H "api-key: $AZURE_OPENAI_API_KEY" \
  -F "purpose=fine-tune" -F "file=@validation.jsonl"

Reference: Files API.

Replace <SUPPORTED_MODEL_ID> and the file IDs, then submit the job with default hyperparameters. The grader matches the exact-answer sample dataset:

curl -X POST "$AZURE_OPENAI_ENDPOINT/openai/v1/fine_tuning/jobs" \
  -H "api-key: $AZURE_OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "<SUPPORTED_MODEL_ID>",
    "training_file": "<TRAINING_FILE_ID>",
    "validation_file": "<VALIDATION_FILE_ID>",
    "method": {
      "type": "reinforcement",
      "reinforcement": {
        "grader": {
          "name": "medmcqa_ans_grader",
          "type": "string_check",
          "input": "{{item.reference_answer}}",
          "reference": "{{sample.output_text}}",
          "operation": "eq"
        }
      }
    },
    "suffix": "my-rft-model"
  }'

Reference: v1 fine-tuning REST API.

For structured output, add your response-format object as method.reinforcement.response_format and replace the grader with one that evaluates that JSON output. Don't use the MedMCQ exact-text grader unchanged.

Use the Azure Developer CLI

Install Azure Developer CLI version 1.22.1 or later, then install the fine-tuning extension and sign in:

azd ext install azure.ai.finetune
azd auth login

Reference: Azure Developer CLI.

Download the YAML configuration and its data or grader files for your method:

Method	Configuration samples
SFT	Supervised fine-tuning.
DPO	Direct preference optimization.
RFT	Reinforcement fine-tuning and graders.

Review the model, method, data paths, and training type. From the configuration directory, initialize the project, submit the job, and inspect its status:

azd ai finetuning init -e <project-endpoint>
azd ai finetuning jobs submit -f <path-to-job-yaml>
azd ai finetuning jobs show -i <job-id>

Use a project endpoint in the form https://<account>.services.ai.azure.com/api/projects/<project>. Replace <job-id> with the ID returned by submission.

Reference: Fine-tuning CLI samples and commands.

Pause, resume, or cancel

Run lifecycle commands only when your model, method, and job state support them. Pause and resume aren't available for every model or method.

azd ai finetuning jobs pause -i <job-id>
azd ai finetuning jobs resume -i <job-id>
azd ai finetuning jobs cancel -i <job-id>

Reference: Fine-tuning CLI commands.

Review RFT cost management and fine-tuning pricing before starting or resuming a job. Billing and spending-limit behavior depend on your selected model.

Review training metrics

RFT measures the rewards assigned by your grader, not SFT token-prediction accuracy. Compare rewards, grader errors, and reasoning-token use across training and validation.

Metric availability depends on your model and grader configuration:

Metrics	What to look for
Training mean reward.	Average grader reward for the current training batch. Batches differ, so inspect the trend rather than comparing individual steps directly.
Validation mean reward.	Average grader reward over the validation set. Use this more stable signal to compare checkpoints, then inspect held-out responses.
Per-grader rewards.	For combined graders, inspect each grader's scores. A higher overall reward can hide poor performance on one criterion.
Grader error rates.	Check parse errors, missing reference fields, and execution errors. These indicate problems with the output format or grader configuration.
Mean reasoning tokens, when reported.	Compare token use with reward and held-out task quality. More reasoning tokens don't necessarily mean a better model.

In metric events, overall rewards are train_reward_mean and valid_reward_mean under data.scores. Per-grader scores are under data.scores.graders.

Reasoning-token metrics are train_reasoning_tokens_mean and valid_reasoning_tokens_mean under data.usage.samples. Grader errors are under data.errors.graders. These aren't SFT loss or token-accuracy fields.

When the job provides linked automatic evaluations, compare scores and failed responses across runs and checkpoints. High grader rewards alone don't establish that the model solves your task.

For explanations of reward-quality risks, see challenges and limitations.

Monitor the job and select a checkpoint

Inspect training progress before choosing a model or checkpoint to deploy.

In the portal, open the job details:

1. Check the Status and event logs. Jobs can queue before training starts; inspect error details if a job fails.
2. Open Monitor to compare training and validation metrics using the guidance above.
3. Open Checkpoints to inspect available model versions and their metrics. Compare candidates on held-out tasks before choosing one to deploy.

Inspect logs and checkpoints with Python

Use the Python client from your submission example. Replace <JOB_ID> with your job ID. Repeat the status check until the job finishes; don't resubmit a queued job.

Retrieve the status, event logs, available checkpoints, and result-file IDs:

job = client.fine_tuning.jobs.retrieve("<JOB_ID>")
print("Status:", job.status)
print("Error:", job.error)
print("Model:", job.fine_tuned_model)
print("Result files:", job.result_files)

for event in client.fine_tuning.jobs.list_events(job.id).data:
    print(event.created_at, event.message)

checkpoints = client.fine_tuning.jobs.checkpoints.list(job.id)
print(checkpoints.model_dump_json(indent=2))

Reference: Fine-tuning API.

After completion, if the job returns a CSV metrics file, replace <RESULT_FILE_ID> with its ID and download it:

with open("results.csv", "wb") as result_file:
    result_file.write(client.files.content("<RESULT_FILE_ID>").read())

Reference: Files API.

Inspect logs and checkpoints with REST

Use the resource endpoint and key for REST. Replace <JOB_ID> with your job ID. Repeat the status check until the job finishes; don't resubmit a queued job.

Retrieve the job, event logs, and available checkpoints:

JOB_URL="$AZURE_OPENAI_ENDPOINT/openai/v1/fine_tuning/jobs/<JOB_ID>"
curl "$JOB_URL" -H "api-key: $AZURE_OPENAI_API_KEY"
curl "$JOB_URL/events" -H "api-key: $AZURE_OPENAI_API_KEY"
curl "$JOB_URL/checkpoints" -H "api-key: $AZURE_OPENAI_API_KEY"

Reference: Fine-tuning API.

After completion, inspect result_files in the job response. If it includes a CSV metrics file, replace <RESULT_FILE_ID> with its ID and download it:

curl "$AZURE_OPENAI_ENDPOINT/openai/v1/files/<RESULT_FILE_ID>/content" \
  -H "api-key: $AZURE_OPENAI_API_KEY" --output results.csv

Reference: Files API.

Use the Azure Developer CLI job workflow to inspect the job status.

Checkpoints appear as training progresses; a queued job might have none. Inspect the returned checkpoint identifiers and metrics rather than assuming the latest version performs best.

When training succeeds, retain the trained model or your chosen checkpoint. Compare candidates on held-out tasks before selecting one for deployment.

Deploy the model

Use the Foundry portal to deploy the model, regardless of how you submit the training job. Follow Deploy fine-tuned models for your model's supported serving option and inference procedure.

After deployment, use Run evaluations from the Foundry portal to compare candidates on held-out tasks.

For SDK evaluation, see Cloud evaluation with the Microsoft Foundry SDK.

Stop training and clean up

Cancel an unneeded job from its portal view. Delete uploaded files separately through the portal when you no longer need them.

Cancel an unneeded job with client.fine_tuning.jobs.cancel("<JOB_ID>"). Delete unused uploaded files separately with client.files.delete("<FILE_ID>").

Reference: Job cancellation and file deletion.

Cancel an unneeded job with the fine-tuning cancellation API. Delete unused uploaded files separately with the Files API.

Cancel an unneeded job with the CLI commands above. For available cleanup operations, see the fine-tuning CLI reference.

Delete unused deployments with the deployment cleanup procedure. Cancelling training doesn't delete deployments or stop their charges.

Retain any data and job records you need to reproduce the experiment.

Related content

• Countdown RFT workflow.