Greetings!
The performance discrepancy between the development and production environments in Azure Data Factory pipelines is likely due to the difference in concurrent job settings. The production environment is configured to handle 55 concurrent jobs, while the development environment is set to 36. This variation can significantly impact performance, especially when dealing with large data extractions from SAP to Azure Data Storage.
To address this issue, align the concurrent job settings between both environments. Adjust the production environment to match the development environment's configuration or vice versa. This alignment should help achieve consistent performance across both environments.
Additionally, ensure that your Self-Hosted Integration Runtime (SHIR) is optimized and that multiple nodes are actively connecting to distribute the workload efficiently. This can help mitigate any memory issues caused by sequential processing.
It's also important to continue monitoring the pipeline performance after making these adjustments and run further tests to ensure stability. If the issue persists, consider reviewing other environmental factors or configurations that might affect performance.
Resource:
Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
Please do not forget to "up-vote" wherever the information provided helps you, as this can be beneficial to other community members.