Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
If you're a startup founder, CTO, or first engineering hire planning an AI workload, you have more than one way to use GPU-accelerated AI compute in Microsoft Azure. The best choice depends on whether you need inference, fine-tuning, distributed training, or batch processing, as well as the scale and operational ownership your team wants to take on.
AI workloads rarely begin as a single production endpoint. A team might start by experimenting with prompts and open models, build a small evaluation harness, fine-tune a model for a specialized task, or run a non-production copy of a service for internal users. As the product gains traction, the same team may need predictable inference latency, higher throughput, regional deployment, or scheduled training and batch jobs.
Each phase places different demands on AI infrastructure. An experimentation workload values speed and flexibility. A training workload values distributed execution, data movement, checkpoint recovery, and repeatability. A production inference workload values a dependable serving path, predictable performance, and an operating model that lets a small team focus on the product. Batch workloads often prioritize throughput, scheduling, and overall cost efficiency over interactive latency.
AI compute needs across the startup lifecycle
| Product moment | Typical startup use case | What to consider |
|---|---|---|
| R&D / Experimentation | Compare models, prompts, retrieval approaches, or agents. | Keep the environment flexible and measure quality before optimizing for scale. |
| Training and fine-tuning | Adapt an open model to domain data or improve a specialized capability. | Plan for data movement, checkpoints, distributed training, and repeatable jobs. |
| Non-production | Run staging, evaluation, red-team, or internal preview environments. | Separate test demand from production demand and schedule work when practical. |
| Production | Serve an AI feature such as search, copilots, document processing, or recommendations. | Set targets for latency, throughput, reliability, regions, and cost per request. |
Why consider more than one compute path
Startups often default to a single infrastructure pattern and only revisit it after requirements evolve. In practice, experimentation, training, validation, and production have different compute requirements. Microsoft Foundry brings together multiple paths that can fit different stages of product development. Considering these paths early can help you keep the architecture aligned to the work you need to do, without taking on infrastructure responsibilities before they're useful.
For example, a document-intelligence startup might compare open models during exploration, fine-tune a model on representative documents, validate the feature in a staging environment, and then serve it to production customers. The model, traffic shape, and success criteria might change at each step. Treating experimentation, training, non-production, and production as separate workloads makes those changes easier to evaluate.
Tip
Confirm the current service status, supported regions, quotas, and commercial terms for your scenario before you commit.
Choose a path by workload
| Your workload | Option to consider | Use cases | Pricing model |
|---|---|---|---|
| Frontier and open model inference | Azure OpenAI and Microsoft Foundry Models | Azure OpenAI features enterprise-grade access to OpenAI models with security, compliance, networking, and Azure integration. Foundry provides broad model choice across proprietary and open models through a unified platform. Fully managed APIs require no infrastructure management. | Pay-per-token, provisioned throughput billed per PTU for predictable capacity. |
| Open model inference at scale | Fireworks on Microsoft Foundry | Optimized high-scale open model serving, with high throughput and low latency. Supports custom models (BYOW). | Pay-per-token, PTU commitment (fungible with ADM PTUs). |
| Dedicated managed inference GPUs | Managed compute in Foundry (preview) | Reserved inference capacity through managed GPUs for open model inference with predictable performance and workload isolation. | Pay as you go per GPU. |
| Fine-tuning or distributed training | Custom Code + Managed Compute Training (private preview) | Distributed training, fine-tuning, and the choice of managed GPUs or BYO IaaS. | BYO IaaS GPUs (pay as you go or Reserved Instances). |
| Very large-scale training or GPU batch | Dapple partnership | Large-scale training and GPU batch, AKS integration, and an Azure control plane that can be extended to private cloud. | Dapple-led pricing and terms, private offer. |
1. Start with inference needs
Azure OpenAI Service and Microsoft Foundry Models are the recommended starting point when your team wants managed access to frontier and open models through APIs. These services abstract away infrastructure management and provide the fastest path from application development to production deployment.
Fireworks AI on Microsoft Foundry supports catalog and custom open-model inference, including on-demand and provisioned deployment options. Available models, deployment types, regions, accelerator configurations, and compliance characteristics vary. Confirm the current model card and Fireworks on Foundry documentation before selecting a deployment. For more information on the shortest path to run an Azure-hosted application that calls a Fireworks AI model deployed through Microsoft Foundry, see Deploy and serve Fireworks AI models on Microsoft Foundry.
Managed compute in Foundry is designed for open-model inference on dedicated GPU capacity with a managed deployment experience. It's well suited for teams that need predictable performance, workload isolation, and the ability to scale inference workloads without directly managing GPU infrastructure.
Note
Managed compute in Foundry is currently in preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
2. Separate training from inference
Training and inference are different workloads and should be evaluated independently. A deployment optimized for production inference is rarely the best environment for fine-tuning or distributed training. Custom Code + Managed Compute Training, currently in private preview, enables teams to train models using their own code, frameworks, and workflows, combining managed GPUs with BYO IaaS, while using the scale, performance, and governance of Microsoft Foundry.
Tip
A deployment that's effective for inference isn't automatically the best environment for training. Estimate data movement, checkpoint storage, interconnect needs, job duration, and retry behavior separately.
3. Treat large-scale batch as a distinct pattern
Large-scale training and batch processing have different requirements than interactive AI applications. When workloads scale to hundreds of GPUs, scheduling, capacity management, infrastructure operations, and procurement can become larger challenges than model development itself.
Dapple is designed for organizations operating large-scale training or GPU batch workloads and provides an Azure-integrated operating model for managing that scale. Organizations should evaluate Dapple when training or batch requirements exceed the scale or operational model of conventional managed deployments. It's available through Microsoft Marketplace and uses Dapple-led private offer pricing and longer-term contracts.
Dapple also includes AKS integration and an Azure control plane that can be extended to private cloud. This capability matters when your workload spans environments or when you need a more specialized operating model.
What comes next?
As your AI workload grows, the decision shifts from selecting a compute path to managing capacity, reliability, and cost across multiple workloads. Consider a deeper architecture review when your workload has one or more of these growth signals:
- Multiple production models
- Sustained high utilization
- Multi-region deployment requirements
- Training and inference workloads competing for the same capacity
- Strict data residency or compliance requirements
- A need to forecast AI spend by product, feature, or team.
At that stage, begin treating AI infrastructure as a portfolio of workloads rather than a single deployment. Document demand by workload, reserve capacity for critical paths, and measure both cost per request and cost per training run. Resist the temptation to adopt enterprise FinOps tooling early. For most startups, a lightweight budget, tags, and a monthly review are usually sufficient while product requirements and usage patterns continue to evolve.