Azure and Python / Pandas (R)

Question

We have our own Azure (windows) server where we manage everything except OS etc.
We have access to the Azure Portal where we can add other servers and many other cool things but I am not used to that.
I am a Python/anaconda guy so I installed Anaconda on the Azure machine and use the SQL agent and call Python scripts for most other stuff (except pure SQL).
(use CMD-files that call Python for web scraping, web api calls, sftp etc)
I can schedule jobs in the SQL agent and be productive - on the other hand it feels like I'm doing things the OLD way.
Result is often consumed in Power BI.

Is there an "Azure Cloud" way to accomplish the same thing using Azure:s new/cool tools.
I want the power of Python/Anaconda without having to go into dozens of menus to put one little piece of code into production.
Sometimes it feels like Microsoft wants you to do only "web server" or "Machine Learning" stuff with Python and for everything else you should use other tools.
I don't want to be limited in a little box where I can't access the outside world from Python/anaconda.
On the other hand I understand the advantages of using the Azure portal where you could have a better overview ow flows and maybe get more security.
Is there a better (cloud) way to do the things I'm doing ?

PS I also work for another company where they use R and the same question goes for that too ...

Answer 1

Hello @Mikko998 ,
Thanks for the ask and using the Microsoft Q&A platform .
I think you can use Azure data factory( ADF ) for most of the part ( I am in doubt for the web scraping ) . Read about this here .
In ADF you can install extenal libraries , whcih can a blocket for you . The other option should be to use Azure data bricks .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

I found an article on Medium (search and find) :
"Automating Python Based Data Transformations With Azure"
It uses ADF and Pandas.
Maybe that is the way to go ...