Migrate Spark libraries from Azure Synapse to Fabric

Azure Synapse Spark pool offers different feed and custom library options. In Fabric, each runtime comes with a preinstalled set of libraries (built-in libraries). However, based on your scenarios and specific needs, you can also include other libraries using environment libraries and inline libraries. To move Azure Synapse Spark pool libraries to Fabric, use environment libraries.

For Spark library considerations, refer to differences between Azure Synapse Spark and Fabric.

Prerequisites

• If you don’t have one already, create a Fabric workspace in your tenant.
• If you don’t have one already, create an Environment in your workspace.

Option 1: Adding Spark libraries to custom environment

You can move Spark pool libraries to an environment as follows:

1. Open Synapse Studio: Sign-in into Azure. Navigate to your Azure Synapse workspace and open the Synapse Studio.
2. Locate Spark libraries:
   • Go to Manage area and select on Apache Spark pools.
   • Find the Apache Spark pool, select Packages and locate the Spark libraries for the pool.
3. Get Spark libraries: Locate the requirements.txt, environment.yml, or workspace packages installed in the pool. Get the list of installed libraries on the pool.
4. Once you have Spark libraries, add custom Spark libraries to an environment in Fabric. Within your Environment, go to Libraries and add libraries:
   • From Public Libraries, you can upload a .yml file. You can also install libraries using PyPI and Conda.
   • From Custom Libraries, you can install libraries by using .jar/.whl/.tar.gz files.
5. Click on Save and Publish changes.

Learn more on adding Spark libraries to an Environment.

Note

Note that library installation may take some time.

Related content

• Migrate Spark configurations
• Migrate Spark notebooks
• Migrate Spark job definition