How to allocate the required memory?

Question

How to allocate the required memory?

Siddhesh Bhurke 46

Error : cannot allocate vector of size 20.5 Gb Error

I am calling multiple notebooks within one notebook that trains my model and saves the results.

dbutils.notebook.run("../path/to_model_file",   timeout_seconds = 48000,  arguments = {"yearperiod":arg_period})

`

I am working on azure databricks and when I run a function it throws this error. My code is written in R

My cluster configuration is -
6.4 (includes Apache Spark 2.4.5, Scala 2.11)

worker type:
standard_ds4_v2 28gb 8 cores 1.5DBU 2-8 workers

driver type:
standard_ds3_v2 14gb 4 cores 0.75DBU

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2021-01-12T03:02:33.82+00:00

@Siddhesh Bhurke Thanks for using Microsoft Q&A !!
I am checking internally with the products team on this issue and get back to you at the earliest.

1 answer

Your answer

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2021-01-12T03:02:33.82+00:00

@Siddhesh Bhurke Thanks for using Microsoft Q&A !!
I am checking internally with the products team on this issue and get back to you at the earliest.

Answer 1

Saurabh Sharma 23,851 Microsoft Employee Moderator

@Siddhesh Bhurke I have talked to the products team and as this would require deeper inspection of code and environment it has been recommended to create a support ticket.

In case you have any restrictions creating a support ticket then please send an email to azcommunity@microsoft.com with your subscription id with this Q&A thread link and subject as "Attn: Saurabh' and I will help providing a free support ticket for you.

Also, provide the support ticket number here so that I can work internally to expedite the resolution.

Thanks
Saurabh

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2021-01-20T16:17:35.823+00:00

@Siddhesh Bhurke Please let me know if you need any help on creating a support ticket.
Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2021-01-22T19:49:06.697+00:00

@Siddhesh Bhurke Following up to know if you need help on this issue.

Share via

How to allocate the required memory?

1 answer

Your answer