Share via

Agent playground billing with an outrageous unexpected charge

florent noirot 30 Reputation points
2025-09-15T14:04:00.91+00:00

Hi,

Yesterday I was playing around in the Agent playground of the Azure AI foundry. I created 3 agents, 1 coordinating 2 other agents. But I wanted to test the retrieval of data of one of the subagent from a keyvault containing only one secret. I set up a custom tool with the schema necessary for the agent to do this and hold and behold it works....except it uses 30k+ token. That seemed excessive so I ask a friend to do the same thing and it cost him 3k token. That's my first problem

The next morning I'm curious to see where my current bill stands at.....and I see 10 euros over night!

I had a very small chat of about 10 exchanges which cost a massive 500k token....but GPT4.1 cost about 3euros/M so there is something wrong somewhere .

So I go to the cost management board and what do I find only for yesterday is a 10 euros charge for "Machine Learning - Safety Evaluation Input Token". What is that? I never asked for a safety evaluation of the input, plus it is not mentioned anywhere during the GUI process of setting up a foundry environment, agents... Am I missing something?

Can someone explain to me what is it exactly and how it can be disabled? Because of the cost of a very casual chat is this, I dont see why anyone would use azure. This is 20x the cost of using the model.

Foundry Tools
Foundry Tools

Formerly known as Azure AI Services or Azure Cognitive Services is a unified collection of prebuilt AI capabilities within the Microsoft Foundry platform

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Manas Mohanty 16,935 Reputation points Microsoft External Staff Moderator
    2025-09-24T01:11:35.5733333+00:00

    Hi florent noirot , Sam Wynn and all

    Got confirmation from product group, you can disable "safety evaluation tokens" by unchecking "Risk and Safety" feature as show in screenshot.

    Hope it reduces the unexpected cost during POCs.

    User's image

    Thank you

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments
  2. florent noirot 30 Reputation points
    2025-09-15T17:46:26.5233333+00:00

    I figured it out. First it makes no sense that evaluation is 2c per 1000 token, when everything is priced per million tokens. This comes out at 20euro/million when chatgpt4.1 at 3euros. 6 times more expensive than the actual LLM to evaluate the safety. They either made a mistake or it's an actual rip-off.

    Second, when I created the resource AI azure foundry, it enabled it by default. no mention of it nowhere. You expect to pay 3 euros per millions, nah it's actually 23 euros per million. That is unacceptable. That a company like Microsoft goes into such practice is unbelievable. No wonder their azure revenues are up if they engage in such practice.

    So to disable it, there is an icon right of "docs" that will open a panel where you will be able to disable this "safety" feature

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments
  3. Azar 31,720 Reputation points MVP Volunteer Moderator
    2025-09-15T17:09:07.4266667+00:00

    Hi there florent noirot

    Thanks for using QandA platform

    I guess foundry doesn’t just send your prompt to the model. It may also run safety evaluations on your inputs/outputs (checking for harmful or disallowed content) and those evaluations are billed separately as “Machine Learning – Safety Evaluation Input Token.

    Right now there’s no toggle in the UI to disable safety evaluations for playground sessions. If you want predictable costs, the better option is to deploy your agent setup into your own app or Function and call the model endpoints directly, where you only pay for the tokens used. The playground is great for prototyping but not cost-optimized.

    I’d recommend opening a support request if you want to clarify the billing breakdown or confirm if your subscription has any optional evaluation features enabled,

    If this helps kindly accept the answer thanks much.

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.