quota increase Synapse pipelines in a Synapse workspace

Chioaru, Monalisa 1 Reputation point
2025-02-04T09:18:20.8566667+00:00

Hello,

I need help to understand how workspace limit can be extend , associated process and cost.

As we know with Default azure synapse configuration we have below given limit of artifacts

Resource - Synapse pipelines in a Synapse workspace

Default limit - 800

Resource - Total number of entities, such as pipelines, data sets, triggers, linked services, Private Endpoints, and integration runtimes, within a workspace Default limit - 5,000

https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits

How much time Quota Increase process takes/ Do We have any limit for increasing the Quota?

If not Quota Increase option then what is the other approach ?

If a possibility is to go by new Workspace, how new workspace and existing one will handle reusable artifacts and Devopps?

Thank you

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,178 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Vinodh247 27,551 Reputation points MVP
    2025-02-10T01:03:52.8833333+00:00

    Hi ,

    Thanks for reaching out to Microsoft Q&A.

    Quota Increase for Azure Synapse Artifacts

    1. Default Limits:
      • Synapse Pipelines per workspace: 800.
        • Total number of entities per workspace (pipelines, datasets, linked services, triggers, etc.): 5,000.
    2. Quota Increase Process:
      • You can request a quota increase through the Azure portal:
      • Navigate to Help + Support > New Support Request.
      • Choose the appropriate Subscription and service (Synapse Analytics).
      • Specify the resource type (Synapse workspace) and the specific quota (pipelines, artifacts).
      • Submit the request, explaining your use case.
      • Timeframe: Typically, Microsoft responds to such requests within 1 to 3 business days, but the actual time depends on the size of the increase and Microsoft’s internal approval process.
      • Limit on Increase: There is no published hard limit on the maximum quotas, but extremely high requests may require justification and alignment with azure's backend engineering limits.

    Alternative Approaches if Quota Increase is Not an Option

    1. Create a New Synapse Workspace:
    • If your quota increase request is denied or insufficient, you can create another Synapse workspace to distribute your pipelines and artifacts.
      • Reusable Artifacts:
        • Linked Services: These are specific to each workspace and cannot be shared directly. You’ll need to duplicate them across workspaces or use centralized configuration management (azure keyvault for connection strings and secrets).
        • Integration Runtimes: Can be reused if configured globally (SHIR Runtime).
        • Data Sharing:
          - Use Azure Data Lake Storage or other shared storage solutions to facilitate data movement between workspaces.
          
                - Use Synapse Link or Azure Data Share for seamless sharing across workspaces.
          
                - DevOps Integration:
          
                      - Configure your DevOps pipelines to support multi-workspace deployment.
          
                            - Use template-based deployments with ARM templates or Bicep to ensure consistency across workspaces.
          
                            - Downsides: Managing multiple workspaces increases operational overhead, especially for monitoring and governance.
          
    1. Refactor and Optimize:
    • Check if any pipelines or artifacts can be consolidated to reduce the number of pipelines/entities.
      • Explore reusable pipeline components (Parameterized Pipelines) to avoid duplication of similar pipelines.

    Quota Increase Costs:

    • There is no direct cost for a quota increase, but increasing quotas may lead to higher costs if the resources (IR, pipelines) consume more compute or data movement.

    Key Considerations for Multiple Workspaces

    1. Artifact Versioning and Consistency:
      • Use a centralized source control system, like Azure Repos or GitHub, to manage pipeline code and ensure consistency between workspaces.
    2. Inter-Workspace Collaboration:
      • Use shared Azure services (KV, Data Lake) to manage configuration and data centrally.
      • Implement clear governance policies for managing pipelines across workspaces.
    3. Governance and Monitoring:
      • Leverage Azure Monitor or Log Analytics for unified monitoring of all workspaces.

    Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.