Unexpected behaviour with %%time magic command in Synapse spark

Grant, Josh 1 Reputation point
2021-03-15T16:15:18.23+00:00

The Synapse documentation states that the %%time magic command can be used in Spark notebooks in Synapse. This does not seem to be the case when calling those notebooks from a pipeline. Here's what I'm experiencing, using PySpark notebooks in Synapse pipelines:

  1. If a notebook cell starts with %%time, it works in the notebook editor but the entire cell is bypassed when run from a pipeline. Bypassed, as in the output indicates that the prior cell runs, then the following cell, but the cell with %%time is just missing from the output!
  2. If a notebook cell contains %time (just a single %), the notebook editor highlights it as incorrect syntax, but it still runs OK. When run from a pipeline, the cell generates an error.

MS, is this a bug, or is the Synapse documentation wrong and %%time is not supposed to work?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,368 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Saurabh Sharma 23,676 Reputation points Microsoft Employee
    2021-03-18T19:17:45.613+00:00

    @Grant, Josh I have received confirmation from the products team that -

    • We currently do not support %%time and %time in synapse pipeline yet. Products team is working to update the documentations to make it more clear for all users.
    • Also, %time is a cell magic, the correct syntax is to put the cell statement in the same line with %time, like below:
      %time 2*4 79250-image.png

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.