Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Generative Artificial Intelligence Operations, or GenAIOps (sometimes called LLMOps), describes the operational practices and strategies for managing large language models (LLMs) in production. This article provides guidance on how to advance your capabilities in GenAIOps, based on your organization's current maturity level.
Use the descriptions below to find your GenAIOps Maturity Model ranking level. These levels provide a general understanding and practical application level of your organization. The guidelines provide you with helpful links to expand your GenAIOps knowledge base.
Tip
Use the GenAIOps Maturity Model Assessment to determine your organization's current GenAIOps maturity level. The questionnaire helps you understand your organization's current capabilities and identify areas for improvement.
Level 1 - initial
Tip
Score from GenAIOps Maturity Model Assessment: initial (0-9).
Description: Your organization is at the initial foundational stage of GenAIOps maturity. You're exploring the capabilities of LLMs but haven't yet developed structured practices or systematic approaches.
Begin by familiarizing yourself with different LLM APIs and their capabilities. Next, start experimenting with structured prompt design and basic prompt engineering. Review Microsoft Learn articles as a starting point. Taking what you've learned, discover how to introduce basic metrics for LLM application performance evaluation.
Suggested references for level 1 advancement
- Foundry model catalog
- Explore the Microsoft Foundry portal Model Catalog
- Introduction to Prompt Engineering
- Prompt Engineering Techniques
- System Message Framework
- Prompt Flow in Foundry portal
- Evaluate GenAI Applications with Foundry
- GenAI Evaluation and Monitoring Metrics with Foundry
To better understand GenAIOps, consider available Microsoft Learning courses and workshops:
Level 2 - defined
Tip
Score from GenAIOps Maturity Model Assessment: maturing (10-14).
Description: Your organization has started to systematize LLM operations, with a focus on structured development and experimentation. However, there's room for more sophisticated integration and optimization.
To improve your capabilities and skills, learn how to develop more complex prompts and integrate them effectively into applications. Implement a systematic approach for LLM application deployment, possibly exploring CI/CD integration. Begin employing more advanced evaluation metrics like groundedness, relevance, and similarity. Focus on content safety and ethical considerations in LLM usage.
Suggested references for level 2 advancement
- Take our step-by-step workshop to elevate your GenAIOps practices
- Prompt Flow in Foundry portal
- How to Build with Prompt Flow
- Deploy a Flow as a Managed Online endpoint for Real-Time Inference
- Integrate Prompt Flow with GenAIOps
- GenAI Evaluation with Foundry
- GenAI Evaluation and Monitoring Metrics
- Azure Content Safety
- Responsible AI Tools and Practices
Level 3 - managed
Tip
Score from GenAIOps Maturity Model Assessment: maturing (15-19).
Description: Your organization is managing advanced LLM workflows with proactive monitoring and structured deployment strategies. You're close to achieving operational excellence.
To expand your knowledge base, focus on continuous improvement and innovation in your LLM applications. Enhance your monitoring strategies with predictive analytics and comprehensive content safety measures. Learn to optimize and fine-tune your LLM applications for specific requirements. Strengthen your asset management strategies through advanced version control and rollback capabilities.
Suggested references for level 3 advancement
- Fine-tuning with Azure ML Learning
- Model Customization with Fine-tuning
- GenAI Model Monitoring
- Elevate LLM Apps to Production with GenAIOps
Level 4 - optimized
Tip
Score from GenAIOps Maturity Model Assessment: optimized (20-28).
Description: Your organization demonstrates operational excellence in GenAIOps. You have a sophisticated approach to LLM application development, deployment, and monitoring.
As LLMs evolve, maintain your cutting-edge position by staying updated with the latest LLM advancements. Continuously evaluate the alignment of your LLM strategies with evolving business objectives. Foster a culture of innovation and continuous learning within your team. Share your knowledge and best practices with the wider community to establish thought leadership in the field.