ASP.NET
A set of technologies in the .NET Framework for building web applications and XML web services.
3,288 questions
You would use the headless mode. Pick a driver that matches your requirements.
Client Sraper - Azure MVC
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 16%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JStus
41
Reputation points
I think the answer is no, but I'll ask anyway. I've created a simple MVC application (hosted in Azure) that uses Selenium to scrap a specific website. Essentially, when running on my local computer, the program launches my Chrome and collects some information from another website. Of course it doesn't work when I deploy my app to Azure. Is there a way to scrap a website using javascript? I have seen several examples using
require('selenium-webdriver')
but I couldn't get it to work.
3 answers
Sort by: Most helpful
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 61%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bruce (SqlWork.com) 56,931 Reputation points2022-09-09T16:30:35.58+00:00 -
JStus 41 Reputation points
2022-09-09T16:55:58.8+00:00 I am, problem is it is trying to open the browser on the server side. I am getting error - "no chrome binary at c:\Program Files....."
Here is my code:
var options = new ChromeOptions(){ BinaryLocation = "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe" }; options.AddArguments(new List<string>() { "headless", "disable-gpu" }); var browser = new ChromeDriver(options);This is why I was thinking maybe there is a way to mimic something like that using JavaScript,, maybe there is some NuGet package that can do it?
Sign in to comment -
-
VenkateshDodda-MSFT 18,851 Reputation points Microsoft Employee2022-09-14T12:04:49.34+00:00 @JStus Thank you for reaching out to Microsoft Q&A, based on the above information shared we have understood that you want to scrap a website from your application (hosted on azure) could you please help us whether you are leveraging the Azure App service or Azure VM to host you application?
- If you are hosting your application using Azure app service as @Bruce (SqlWork.com) suggested you need to use headless mode to scrap a website and Whatever Chrome or Chromium with headless or non-headless, they all require GDI support. However, on Azure App Services on Windows, it conflicts with Win32k.sys (User32/GDI32) Restrictions, as the figure below.
- And other frameworks like PhantomJS/Selenium also be restricted by it, see below.
So, you cannot use chromium within Azure WebApp on Windows.
Alternatively, Checkout this blog where anthonychu one of the PM shares insights on [Azure Linux function app] (https://anthonychu.ca/post/azure-functions-headless-chromium-puppeteer-playwright/) to run headless chromium with Puppeteer and playwright.
-
JStus 41 Reputation points
2022-09-15T18:07:56.557+00:00 HI Venkatesh,
It is Azure App service, as I mentioned I am using headless mode, but since the website is trying to open the chrome on the server side - it fails.
I've seen some examples using Node.js but still no luck.Would VM resolve the issue? Will I be able to install Chrome on VM and run the app that will try to open it?
-
VenkateshDodda-MSFT 18,851 Reputation points Microsoft Employee
2022-09-16T09:42:59.463+00:00 @JStus , thanks for your response. Yes, With Azure Virtual Machines (IAAS), you have total control over the configuration of the machine. When using VMs, you're responsible for all server software installation, configuration, maintenance, and operating system patches.
Sign in to comment -
Bruce (SqlWork.com) 56,931 Reputation points
2022-09-20T20:38:13.59+00:00 if you are not tied to coding in C#, you could use node and Playwright (a Microsoft npm package)
https://www.npmjs.com/package/playwright
a sample of azure function (node) using playwright:
https://dotnetthoughts.net/running-playwright-on-azure-functions/
a sample of azure static web app