Synapse Notebook - RunNotebookApiError: Size of notebook content and exit value must be less than 8MB

Question

We have a new Synapse Notebook in which we implemented some asynchronous methods (API calls).

I got a very strange error after trying to run the notebook as part of pipeline (see the error below) and would appreciate any help as I cannot find anything online about the error and how to deal with it.

Context: I import an application from my module and run the app with await as the run() method is asynchronous: async def run(): ...

Notebook content:

from mymodule import App
import asyncio

app = App(env="prd")
a = await app.run()

The notebook cell completes without a problem (has a green checkmark and the 'success' message "Command executed in 1h 20min 9sec 49ms on 12:05:12 PM, ....").

However, the notebook then shows an error in the pipeline afterwards and leads to a crash. The notebook in the pipeline gives me the following error:

Operation on target MY_NOTEBOOK_NAME failed: RunNotebookApiError: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot')

Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.

--> ArgumentOutOfRangeException: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot')

Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.

Answer 1

When running these notebooks as part of a Synapse pipeline, the results and the notebook's state are captured and passed back to the pipeline.

The error message you're seeing indicates that the output/state of your notebook exceeds the 8MB size limit. This can happen if:

1. The notebook contains a large amount of code, markdown, and other contents.
2. The output generated by your notebook (e.g., dataframes, graphs, or other outputs) is substantial.

You need to delve into the run() method in your App class. Check what this method is doing. Does it fetch a large dataset? Does it produce a substantial amount of logs or other outputs?

If the method does produce outputs, try redirecting them to a file or another storage mechanism instead of returning them. Even if these outputs aren't explicitly printed to the notebook, they could still be captured by Synapse and contribute to the size issue.

Test without **await app.run()**if the method does produce outputs, try redirecting them to a file or another storage mechanism instead of returning them. Even if these outputs aren't explicitly printed to the notebook, they could still be captured by Synapse and contribute to the size issue.