Is it possible to connect an Apache Spark Pool in an Azure Synapse workspace to an artifacts feed in Azure Devops?

Nick Petzold 5 Reputation points
2023-02-02T12:21:09.1533333+00:00

I'm trying to configure an environment that my team are going to use with the Apache Spark pool that we've defined in our Synapse workspace.

We have a number of internal packages that we have built and deployed to our own artefacts feed in Azure Devops and would like to install these within spark pool instances when they are spun up. I know it's possible to download the source file and then add this package to the workspace manually, however they are likely to be updated often, so it would be preferable to be able to install these via a supplied requirements.txt file instead (I know you can use requirements.txt with spark pools, but I can only see a way that this will work with publicly available packages).

Does anyone know how to connect a Spark pool/Synapse workspace to an artefacts feed so all of this can be handled seamlessly?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,401 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. BhargavaGunnam-MSFT 26,316 Reputation points Microsoft Employee
    2023-02-07T13:25:32.95+00:00

    Hello @Nick Petzold ,

    Welcome to the MS Q&A platform.

    A similar question has been discussed here

    I hope this helps. Please let us know if you have any further questions.

    0 comments