Hi ,
Thanks for reaching out to Microsoft Q&A.
Quota Increase for Azure Synapse Artifacts
- Default Limits:
- Synapse Pipelines per workspace: 800.
- Total number of entities per workspace (pipelines, datasets, linked services, triggers, etc.): 5,000.
- Synapse Pipelines per workspace: 800.
- Quota Increase Process:
- You can request a quota increase through the Azure portal:
- Navigate to Help + Support > New Support Request.
- Choose the appropriate Subscription and service (Synapse Analytics).
- Specify the resource type (Synapse workspace) and the specific quota (pipelines, artifacts).
- Submit the request, explaining your use case.
- Timeframe: Typically, Microsoft responds to such requests within 1 to 3 business days, but the actual time depends on the size of the increase and Microsoft’s internal approval process.
- Limit on Increase: There is no published hard limit on the maximum quotas, but extremely high requests may require justification and alignment with azure's backend engineering limits.
Alternative Approaches if Quota Increase is Not an Option
- Create a New Synapse Workspace:
- If your quota increase request is denied or insufficient, you can create another Synapse workspace to distribute your pipelines and artifacts.
- Reusable Artifacts:
- Linked Services: These are specific to each workspace and cannot be shared directly. You’ll need to duplicate them across workspaces or use centralized configuration management (azure keyvault for connection strings and secrets).
- Integration Runtimes: Can be reused if configured globally (SHIR Runtime).
- Data Sharing:
- Use Azure Data Lake Storage or other shared storage solutions to facilitate data movement between workspaces. - Use Synapse Link or Azure Data Share for seamless sharing across workspaces. - DevOps Integration: - Configure your DevOps pipelines to support multi-workspace deployment. - Use template-based deployments with ARM templates or Bicep to ensure consistency across workspaces. - Downsides: Managing multiple workspaces increases operational overhead, especially for monitoring and governance.
- Reusable Artifacts:
- Refactor and Optimize:
- Check if any pipelines or artifacts can be consolidated to reduce the number of pipelines/entities.
- Explore reusable pipeline components (Parameterized Pipelines) to avoid duplication of similar pipelines.
Quota Increase Costs:
- There is no direct cost for a quota increase, but increasing quotas may lead to higher costs if the resources (IR, pipelines) consume more compute or data movement.
Key Considerations for Multiple Workspaces
- Artifact Versioning and Consistency:
- Use a centralized source control system, like Azure Repos or GitHub, to manage pipeline code and ensure consistency between workspaces.
- Inter-Workspace Collaboration:
- Use shared Azure services (KV, Data Lake) to manage configuration and data centrally.
- Implement clear governance policies for managing pipelines across workspaces.
- Governance and Monitoring:
- Leverage Azure Monitor or Log Analytics for unified monitoring of all workspaces.
Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.