Use the Copilot for Data Science and Data Engineering chat panel (preview)

Important

This feature is in preview.

Copilot for Data Science and Data Engineering notebooks is an AI assistant that helps you analyze and visualize data. It works with lakehouse tables, Power BI Datasets, and pandas/spark dataframes, providing answers and code snippets directly in the notebook. The most effective way of using Copilot is to load your data as a dataframe. You can use the chat panel to ask your questions, and the AI provides responses or code to copy into your notebook. It understands your data's schema and metadata, and if data is loaded into a dataframe, it has awareness of the data inside of the data frame as well. You can ask Copilot to provide insights on data, create code for visualizations, or provide code for data transformations, and it recognizes file names for easy reference. Copilot streamlines data analysis by eliminating complex coding.

Note

Azure OpenAI enablement

  • Azure OpenAI must be enabled within Fabric at the tenant level.

Note

If your workspace is provisioned in a region without GPU capacity, and your data is not enabled to flow cross-geo, Copilot will not function properly and you will see errors.

Successful execution of Chat-Magics installation cell

  1. To use the Copilot pane, The installation cell for chat-magics must successfully execute within your Spark session.

    Screenshot showing executed notebook cell.

    Important

    If your Spark session terminates, the context for chat-magics will also terminate, also wiping the context for the Copilot pane.

  2. Verify that all these conditions are met before proceeding with the Copilot Chat Pane.

Open Copilot chat panel inside the notebook

  1. Select the Copilot button on the notebook ribbon

    Screenshot showing Copilot ribbon.

  2. To open Copilot, select the Copilot button at the top of the Notebook.

  3. The Copilot chat panel opens on the right side of your notebook.

  4. A panel opens, to provide overview information and helpful links.

    Screenshot showing Copilot helpful links pane.

Key capabilities

  • AI assistance: Generate code, query data, and get suggestions to accelerate your workflow.
  • Data insights: Quick data analysis and visualization capabilities.
  • Explanations: Copilot can provide natural language explanations of notebook cells, and can provide an overview for notebook activity as it runs.
  • Fixing errors: Copilot can also fix notebook run errors as they arise. Copilot shares context with the notebook cells (executed output) and can provide helpful suggestions.

Important notices

  • Inaccuracies: Potential for inaccuracies exists. Review AI-generated content carefully.
  • Data storage: Customer data is temporarily stored, to identify harmful use of AI.

Getting started with Copilot chat in notebooks

  1. Copilot for Data Science and Data Engineering offers helpful starter prompts to get started. For example, "Load data from my lakehouse into a dataframe", or "Generate insights from data".

    Screenshot showing the Copilot starting prompts.

  2. Each of these selections outputs chat text in the text panel. As the user, you must fill out the specific details of the data you'd like to use.

  3. You can then input any type of request you have in the chat box.

    Screenshot showing the Copilot starting prompts.

Regular usage of the Copilot chat panel

  • The more specifically you describe your goals in your chat panel entries, the more accurate the Copilot responses.
  • You can "copy" or "insert" code from the chat panel. At the top of each code block, two buttons allow input of items directly into the notebook.
  • To clear your conversation, select the Screenshot showing the Copilot clean up prompt. icon at the top to remove your conversation from the pane. It clears the pane of any input or output, but the context remains in the session until it ends.
  • Configure the Copilot privacy settings with the %configure_privacy_settings command, or the %set_sharing_level command in the chat magics library.
  • Transparency: Read our Transparency Note for details on data and algorithm use.