Edit

Use Python for Apache Spark

Microsoft Fabric provides built-in Python support for Apache Spark. Support includes PySpark, which you can use to interact with Spark through familiar Spark or Python interfaces.

You can analyze data by using Python through Spark batch job definitions or with interactive Fabric notebooks. This article provides an overview of developing Spark applications in Microsoft Fabric by using the Python language.

Create and run notebook sessions

Microsoft Fabric notebook is a web interface where you create files that contain live code, visualizations, and narrative text. Notebooks are a good place to validate ideas and use quick experiments to get insights from your data. Data engineers widely use notebooks in data preparation, data visualization, machine learning, and other big data scenarios.

To get started with Python in Microsoft Fabric notebooks, change the primary Language at the top of your notebook by setting the language option to PySpark (Python).

%%pyspark
# Enter your Python code here

You can use multiple languages in one notebook by specifying the language magic command at the beginning of a cell.

To learn more about notebooks in Microsoft Fabric Analytics, see How to use notebooks.

Install packages

Libraries provide reusable code that you can include in your programs or projects. To make partner code or locally built code available to your applications, install a library in-line into your notebook session. Alternatively, your workspace administrator can create an environment, install the library in it, and attach the environment as the workspace default in the workspace setting.

To learn more about library management in Microsoft Fabric, see Manage Apache Spark libraries.

Notebook utilities

NotebookUtils (formerly MSSparkUtils) is a built-in package that helps you easily perform common tasks. Use NotebookUtils to work with file systems, get environment variables, chain notebooks together, and work with secrets. PySpark notebooks support NotebookUtils.

To get started, run the following commands:

import notebookutils
notebookutils.notebook.help()

For more information about the supported NotebookUtils commands, see Use NotebookUtils.

Use Pandas on Spark

The Pandas API on Spark scales your Pandas workload to any size by running it distributed across multiple nodes. If you're already familiar with pandas and want to use Spark for big data, the pandas API on Spark makes you immediately productive.

You can migrate your applications without modifying the code. You can have a single codebase that works both with pandas, for tests and smaller datasets, and with Spark, for production and distributed datasets. You can switch between the pandas API and the Pandas API on Spark easily and without overhead.

Python runtime

The Microsoft Fabric Runtime is a curated environment optimized for data science and machine learning. The Microsoft Fabric runtime offers a range of popular, Python open-source libraries, including libraries like Pandas, PyTorch, scikit-learn, and XGBoost.

Python visualization

The Python ecosystem offers multiple graphing libraries that come with many different features. By default, every Spark instance in Microsoft Fabric contains a set of curated and popular open-source libraries. You can also add or manage other libraries or versions. For more information on library management, see Summary of library management best practices.

To learn more about how to create Python visualizations, see Python visualization.

Related content

  • Last updated on