automate getting html from Windows Authentication secured site

Question

automate getting html from Windows Authentication secured site

Nikolay KARETNIKOV 1

Hello!
There’s this application that is managed via a web portal http://localhost/appname/admin
Site is hosted on IIS. The page requires authentication. IIS Authentication settings are as follows: Windows Authentication “Enabled”, Basic Authentication “Disabled”, ASP.NET Impersonalisation “Disabled”, Anonymous Authentication “Enabled”.
Once I login into the system using an AD account, I can browse the portal and get some data. The aim is to automate getting a data from about 1k pages via either powershell or C# code. The problem is that I can not get through the authentication page.
I’m thinking 2 ways. 1. to authenticate in the code itself, 2. to use already established by browser session. But with almost zero experience, I’m stuck. Would someone suggest a way to handle the situation

Sam Wu-MSFT 7,571 Reputation points Microsoft External Staff

2022-07-18T08:28:55.723+00:00

@Nikolay KARETNIKOV

As far as I know, if you enable both Anonymous Authentication and Windows Authentication, the client will first use anonymous authentication, why you're having trouble getting authenticated?
Nikolay KARETNIKOV 1 Reputation point

2022-07-18T10:48:09.457+00:00

here is more on this. Would you care to read and comment?
https://csharpforums.net/threads/scrape-data-from-website-with-login-c.8223/
Sam Wu-MSFT 7,571 Reputation points Microsoft External Staff

2022-07-19T08:11:41.887+00:00

@Nikolay KARETNIKOV

I checked the content in the link you posted and I found that this does not seem to be an iis problem, and the content in the link does not mention the authentication method you use in iis.