Azure function /automation runbook for hosting long running jobs

Question

Azure function /automation runbook for hosting long running jobs

Azure-learning 56

I am using Azure function for hosting powershell function which is consuming a lot of time.

This function generally takes 60min to complete ,is there any other way we can host a long running powershell based script/function. or we can use Azure function to achieve this with some modification. What is the suggested way, using automation run book /function/webjobs or durable functions?

I tried to create function app in premium plan. still I am getting 503 error. looks like it is related to memory.

<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html xmlns='

5 answers

Your answer

Answer 1

Maxim Sergeev 6,586 Microsoft Employee

Hi @NGaur-3476,

Have you tried to change SKU worker plan? For example, to this one: Four vCPU/14 GB RAM

Workers are available in three fixed sizes: One vCPU/3.5 GB RAM; Two vCPU/7 GB RAM; Four vCPU/14 GB RAM

On the other hand, you could use Azure Automation Account with Hybrid Worker Server roles where you use any size of Azure VMs, especially if you think the problem occurs because of memory.
https://learn.microsoft.com/en-us/azure/automation/extension-based-hybrid-runbook-worker-install?tabs=windows

Azure-learning 56 Reputation points

2022-08-31T05:46:48.333+00:00

Hi @Maxim Sergeev I tried to use Premium plan of function app , I am still getting error response 503 in the response ,however function is executing as per powershell script.
this function will have to take backup of multiple azure vm's until it is done. I tested with just one VM and i am getting the error (below).

<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html xmlns='http://www.w3.org/1999/xhtml'><head><meta content='text/html; charset=utf-8' http-equiv='content-type'/><style type='text/css'>body {font-family:Arial; margin-left:40px; }img { border:0 none; }#content { margin-left: auto; margin-right: auto }#message h2 { font-size: 20px; font-weight: normal; color: #000000; margin: 34px 0px 0px 0px }#message p { font-size: 13px; color: #000000; margin: 7px 0px 0px0px}#errorref { font-size: 11px; color: #737373; margin-top: 41px }</style><title>Service unavailable</title></head><body><div id='content'><div id='message'><h2>Our services aren't available right now</h2><p>We're working to restore all services as soon as possible. Please check back soon.</p></div><div id='errorref'><span>0lvQOYwAAAACBom6P479UR72rZnIaj4MhQk9NMDJFREdFMDkxNwA3ZGIwYWMxYy1iNzZkLTRiYTQtYTE3NS01NTgxNTUxMTEzZDU=</span></div></div></body></html>

I will try with Azure automation as suggested .
Maxim Sergeev 6,586 Reputation points Microsoft Employee

2022-08-31T06:30:01.513+00:00

1) Are you using EP3? (Premium Plan with 14 Gb RAM)
2) Are you sure that the powershell script is expected to complete in 60 minutes?

In the Premium plan, the run duration defaults to 30 minutes to prevent runaway executions. However, you can modify the host.json configuration to make the duration unbounded for Premium plan apps. When set to an unbounded duration, your function app is guaranteed to run for at least 60 minutes.

Answer 2

Azure-learning 56

@Maxim Sergeev yes , I think just function is not appropriate . because powershell scripts (taking backup ) will consume lot of time and in longer run will create performance issue and will be costly. Is there any suggestion where we can add eventhub or any middle ware can retain data for certain amount and use different function to process it later asyn .

PCould you please help me if there is any such article/suggestion on this approach .

Maxim Sergeev 6,586 Reputation points Microsoft Employee

2022-08-31T17:47:16.3+00:00

@NGaur-3476, it depends on what and how you want to achieve it.
If you expect to run one huge batch with long-running workflows - I recommend using dedicated instances like Azure VM to execute your scripts. But if could control the flow and split the backup task to multiple instances, you could leverage Function without any problems, but it requires to re-engineer your solution.
So, summarizing the above, I would suggest switching from Functions to Azure Automation Hybrid Worker Servers, where you will be able to run your flows without limitations (except Powershell or VMs limitations themselves), if your script runs successfully from your laptop, it should be executed from Hybrid Worker Server successfully too.

Answer 3

@NGaur-3476, it depends on what and how you want to achieve it.
If you expect to run one huge batch with long-running workflows - I recommend using dedicated instances like Azure VM to execute your scripts. But if could control the flow and split the backup task to multiple instances, you could leverage Function without any problems, but it requires to re-engineer your solution.
So, summarizing the above, I would suggest switching from Functions to Azure Automation Hybrid Worker Servers, where you will be able to run your flows without limitations (except Powershell or VMs limitations themselves), if your script runs successfully from your laptop, it should be executed from Hybrid Worker Server successfully too.

Answer 4

Maxim Sergeev 6,586 Microsoft Employee

Please don't hesitate to mark the post as answered if it is. This helps us and our community.

Answer 5

Hi @NGaur-3476 ,

Thanks for reaching out to Q&A forum.

For long running functions, I would suggest you to use durable functions as they also have powershell support.
https://learn.microsoft.com/en-us/azure/azure-functions/durable/quickstart-powershell-vscode

When the functions is long running, please make sure that it is not hitting the memory and cpu limit. You can confirm this by running the "Function app down or reporting errors" detector available in the Daignose and solve problems blade. Please note that, memory related issues can cause 503 errors.

Feel free to reach out to me if you have any questions.

Share via

Azure function /automation runbook for hosting long running jobs

5 answers

Your answer