Synapse Notebook - RunNotebookApiError: Size of notebook content and exit value must be less than 8MB

Question

We have a new Synapse Notebook in which we implemented some asynchronous methods (API calls).

I got a very strange error after trying to run the notebook as part of pipeline (see the error below) and would appreciate any help as I cannot find anything online about the error and how to deal with it.

Context: I import an application from my module and run the app with await as the run() method is asynchronous: async def run(): ...

Notebook content:

from mymodule import App
import asyncio

app = App(env="prd")
a = await app.run()

The notebook cell completes without a problem (has a green checkmark and the 'success' message "Command executed in 1h 20min 9sec 49ms on 12:05:12 PM, ....").

However, the notebook then shows an error in the pipeline afterwards and leads to a crash. The notebook in the pipeline gives me the following error:

Operation on target MY_NOTEBOOK_NAME failed: RunNotebookApiError: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot')

Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.

--> ArgumentOutOfRangeException: Size of notebook content and exit value must be less than 8MB. Please try to write large outputs to files (Parameter 'resultSnapshot')

Actual value was Microsoft.Analytics.SynapseNotebookExecService.Models.RunNotebookSnapshot.

Answer 1

When running these notebooks as part of a Synapse pipeline, the results and the notebook's state are captured and passed back to the pipeline.

The error message you're seeing indicates that the output/state of your notebook exceeds the 8MB size limit. This can happen if:

1. The notebook contains a large amount of code, markdown, and other contents.
2. The output generated by your notebook (e.g., dataframes, graphs, or other outputs) is substantial.

You need to delve into the run() method in your App class. Check what this method is doing. Does it fetch a large dataset? Does it produce a substantial amount of logs or other outputs?

If the method does produce outputs, try redirecting them to a file or another storage mechanism instead of returning them. Even if these outputs aren't explicitly printed to the notebook, they could still be captured by Synapse and contribute to the size issue.

Test without **await app.run()**if the method does produce outputs, try redirecting them to a file or another storage mechanism instead of returning them. Even if these outputs aren't explicitly printed to the notebook, they could still be captured by Synapse and contribute to the size issue.

Answer 2

Hello @Victor Seifert

The error message you're seeing indicates that the size of the notebook content and exit value must be less than 8MB. This error can occur when the output of a notebook cell is too large to be returned as a result snapshot.

One way to resolve this issue is to write large outputs to files instead of returning them as a result snapshot. You can do this by using the %%capture magic command in your notebook cell to capture the output and write it to a file. Here's an example:

%%capture captured_output
# your code here

with open('output.txt', 'w') as f:
    f.write(captured_output.stdout)

This will capture the output of your code and write it to a file called output.txt. You can then read the contents of this file in a subsequent notebook cell or write it to an ADLS.

Another option is to split your notebook into smaller cells to reduce the size of the output. You can also try to optimize your code to reduce the amount of output it generates.

I hope this helps! Let me know if you have any further questions.