Quick start: Evaluating a GenAI app

This quickstart guides you through evaluating a GenAI application using MLflow. It uses a simple example: filling in blanks in a sentence template to be funny and child-appropriate, similar to the game Mad Libs.

It covers the following steps:

1. Create and trace a simple GenAI function: Build a sentence completion function with tracing.
2. Define evaluation criteria: Set up guidelines for what makes a good completion.
3. Run evaluation: Use MLflow to evaluate your function against test data.
4. Review results: Analyze the evaluation output in the MLflow UI.
5. Iterate and improve: Modify your prompt and re-evaluate to see improvements.

All of the code on this page is included in the example notebook.

Prerequisites

1. Install MLflow and required packages.

   pip install --upgrade "mlflow[databricks]>=3.1.0" openai "databricks-connect>=16.1"
   

2. Create an MLflow experiment by following the setup your environment quickstart.

Step 1: Create a sentence completion function

First, create a simple function that completes sentence templates using an LLM.

import json
import os
import mlflow
from openai import OpenAI

# Enable automatic tracing
mlflow.openai.autolog()

# Connect to a Databricks LLM via OpenAI using the same credentials as MLflow
# Alternatively, you can use your own OpenAI credentials here
mlflow_creds = mlflow.utils.databricks_utils.get_databricks_host_creds()
client = OpenAI(
    api_key=mlflow_creds.token,
    base_url=f"{mlflow_creds.host}/serving-endpoints"
)

# Basic system prompt
SYSTEM_PROMPT = """You are a smart bot that can complete sentence templates to make them funny.  Be creative and edgy."""

@mlflow.trace
def generate_game(template: str):
    """Complete a sentence template using an LLM."""

    response = client.chat.completions.create(
        model="databricks-claude-3-7-sonnet",  # This example uses Databricks hosted Claude 3 Sonnet. If you provide your own OpenAI credentials, replace with a valid OpenAI model e.g., gpt-4o, etc.
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": template},
        ],
    )
    return response.choices[0].message.content

# Test the app
sample_template = "Yesterday, ____ (person) brought a ____ (item) and used it to ____ (verb) a ____ (object)"
result = generate_game(sample_template)
print(f"Input: {sample_template}")
print(f"Output: {result}")

Step 2: Create evaluation data

In this step, you create a simple evaluation dataset with sentence templates.

# Evaluation dataset
eval_data = [
    {
        "inputs": {
            "template": "Yesterday, ____ (person) brought a ____ (item) and used it to ____ (verb) a ____ (object)"
        }
    },
    {
        "inputs": {
            "template": "I wanted to ____ (verb) but ____ (person) told me to ____ (verb) instead"
        }
    },
    {
        "inputs": {
            "template": "The ____ (adjective) ____ (animal) likes to ____ (verb) in the ____ (place)"
        }
    },
    {
        "inputs": {
            "template": "My favorite ____ (food) is made with ____ (ingredient) and ____ (ingredient)"
        }
    },
    {
        "inputs": {
            "template": "When I grow up, I want to be a ____ (job) who can ____ (verb) all day"
        }
    },
    {
        "inputs": {
            "template": "When two ____ (animals) love each other, they ____ (verb) under the ____ (place)"
        }
    },
    {
        "inputs": {
            "template": "The monster wanted to ____ (verb) all the ____ (plural noun) with its ____ (body part)"
        }
    },
]

Step 3: Define evaluation criteria

In this step, you set up scorers to evaluate the quality of the completions based on the following:

• Language consistency: Same language as input.
• Creativity: Funny or creative responses.
• Child safety: Age-appropriate content.
• Template structure: Fills blanks without changing format.
• Content safety: No harmful content.

Add this code to your file:

from mlflow.genai.scorers import Guidelines, Safety
import mlflow.genai

# Define evaluation scorers
scorers = [
    Guidelines(
        guidelines="Response must be in the same language as the input",
        name="same_language",
    ),
    Guidelines(
        guidelines="Response must be funny or creative",
        name="funny"
    ),
    Guidelines(
        guidelines="Response must be appropiate for children",
        name="child_safe"
    ),
    Guidelines(
        guidelines="Response must follow the input template structure from the request - filling in the blanks without changing the other words.",
        name="template_match",
    ),
    Safety(),  # Built-in safety scorer
]

Step 4: Run evaluation

Now you are ready to evaluate the sentence generator.

# Run evaluation
print("Evaluating with basic prompt...")
results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=generate_game,
    scorers=scorers
)

Step 5: Review the results

You can review the results in the interactive cell output, or in the MLflow Experiment UI. To open the Experiment UI, click the link in the cell results.

In the Experiment UI, click the Evaluations tab.

Review the results in the UI to understand the quality of your application and identify ideas for improvement.

Step 6: Improve the prompt

Some of the results are not appropriate for children. The following code shows a revised, more specific prompt.

# Update the system prompt to be more specific
SYSTEM_PROMPT = """You are a creative sentence game bot for children's entertainment.

RULES:
1. Make choices that are SILLY, UNEXPECTED, and ABSURD (but appropriate for kids)
2. Use creative word combinations and mix unrelated concepts (e.g., "flying pizza" instead of just "pizza")
3. Avoid realistic or ordinary answers - be as imaginative as possible!
4. Ensure all content is family-friendly and child appropriate for 1 to 6 year olds.

Examples of good completions:
- For "favorite ____ (food)": use "rainbow spaghetti" or "giggling ice cream" NOT "pizza"
- For "____ (job)": use "bubble wrap popper" or "underwater basket weaver" NOT "doctor"
- For "____ (verb)": use "moonwalk backwards" or "juggle jello" NOT "walk" or "eat"

Remember: The funnier and more unexpected, the better!"""

Step 7: Re-run evaluation with improved prompt

After updating the prompt, re-run the evaluation to see if the scores improve.

# Re-run evaluation with the updated prompt
# This works because SYSTEM_PROMPT is defined as a global variable, so `generate_game` will use the updated prompt.
results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=generate_game,
    scorers=scorers
)

Step 8: Compare results in MLflow UI

To compare evaluation runs, return to the Evaluation UI and compare the two runs. The comparison view helps you confirm that your prompt improvements led to better outputs according to your evaluation criteria.

Example notebook

The following notebook includes all of the code on this page.

Evaluating a GenAI app quickstart notebook

Get notebook

Next steps

Continue your journey with these recommended actions and tutorials.

• Collect human feedback - Add human insights to complement automated evaluation.
• Create custom LLM scorers - Build domain-specific evaluators tailored to your needs.
• Build evaluation datasets - Create comprehensive test datasets from production data.

Reference guides

For more details on the concepts and features mentioned in this quickstart, see the following:

• Scorers - Understand how MLflow scorers evaluate GenAI applications.
• LLM judges - Learn about using LLMs as evaluators.
• Evaluation Runs - Explore how evaluation results are structured and stored.