Azure Web Application with computationally intenstive tasks in Dask and Tensorflow

Question

Hello,

I'm developing a data analysis tool for the processing of data from Hydrogen-Deuterium exchange mass spectrometry. We would like to accompany our publication with a deployment of the code on Microsoft Azure so that other researchers can quickly and easily try our software without needing to install it.

The application is written in Python and we use the package Panel to create a web interface. Although I have not yet done so myself I've found instructions on how to deploy these panel dashboards on microsoft azure, so I think this step won't be a problem.
The main difficulty is that the webapp has two computationally intensive tasks, which have high peak CPU usage but are sparse, as I expect that if 3 users use the app simultaneously it will be a rare high amount of users while mostly no user will be using the app. So I'm wondering how to properly deploy this on Azure, where we can pay per CPU/GPU cycle for the computationally intensive tasks and keep the hardware requirements for the web interface part minimal.

The first computationally intensive task is a normal CPU fitting procedure. I've already separated this from the main web interface by using Dask, which according to their docs I can deploy on microsoft azure using Kubernetes and Helm. I this plan a sensible approach to doing this? Is it indeed necessary to separate both or can I configure Azure differently to deal with this?

The second computationally intensive task is a tensorflow fit, where I've repurposed the neural network architecture to solve my specific problem. The memory requirement for this step is quite low and I run this on the GPU of my laptop (2060) which then completes in a reasonable time (typ. 5 minutes). How can I best do tensorflow (GPU) jobs and submit then remotely? Should I also schedule this using Dask?

Perhaps I should use Databricks?
This is my first time playing around with Azure, so any general pointers are welcome. So far the plethora of options is quite overwhelming.

Answer 1

Thank you for your question @JochemSmit-5263. This is a wide reaching question so I will provide a high level response. If you have further questions, I would suggest making additional posts and ensure you tag the associated products in those posts.

I would suggest that you maybe break out and look into using multiple Azure products.

1. Put the Panel package on an Azure Web App. You can run the web app at the B1 tier to help keep costs down (S1 tier or higher is recommended for production apps but if you're only averaging 3 visitors at a time, you probably don't need that much power. See the below links for Azure Web Apps overview and pricing page. Azure Web Apps are charged for per minute usage and not per CPU/GPU instruction. Basically, you're billed for time used, not resources used. There is not really a straight forward way to scale your web site down from the tier you selected unless you manually scale it up and down when needed.
2. Azure Web Apps are great for hosting web applications and performing computational tasks but if you're looking for heavy CPU and GPU usage, I would recommend offloading that work into an Azure VM and have the web app communicate with the VM. Something like the N-series VMs provide GPUs for your use. This could also likely handle the CPU intensive task as well.

Regarding the Dask questions, I would hit up their community on dask.org as I'm not super familiar with it nor is anyone else on here likely to have the Dask knowledge you deserve to hear from.

Azure Web Apps Overview: https://learn.microsoft.com/en-us/azure/static-web-apps/
Azure Web Apps Pricing: https://azure.microsoft.com/en-us/pricing/details/app-service/windows/
Azure Web App Features per tier: https://azure.microsoft.com/en-us/pricing/details/app-service/plans/
Azure VMs: https://azure.microsoft.com/en-us/pricing/details/virtual-machines/series/