download file from https with credentials doesn't contain data

Question

download file from https with credentials doesn't contain data

Stacy Bell 41

This downloads an online file but the file has the website's html code instead of data. How do you get the file to have data instead of html code?

        var url = string.Format(@"https://www.test.com/Monthly%20Report");
        WebClient webClient = new WebClient();
        webClient.Credentials = new NetworkCredential("userName", "Password123");
        webClient.Headers.Add("cookie", "");
        webClient.DownloadFile(url, @"C:\Users\me\Downloads\Monthly Report.csv");

P a u l 10,766 Reputation points

2021-12-30T00:13:05.367+00:00

Are you able to access the file in the browser?
Stacy Bell 41 Reputation points

2021-12-30T00:19:18.607+00:00

In the app you need to select a date range then click a download button. I'm trying to do that programmatically. The first line should be this.

var url = string.Format(@"https://www.test.com/Monthly%20Report?start%20date=1/1/2021&end%20date=1/15/2021");
P a u l 10,766 Reputation points

2021-12-30T00:22:03.177+00:00

If you're clicking a button could it be performing a POST request instead of GET?

Stacy Bell 41

Help

        HttpWebRequest httpRequest = (HttpWebRequest)
        WebRequest.Create("https://www.test.com/Monthly%20Report?start%20date=1/1/2021&end%20date=1/15/2021");
        httpRequest.Method = WebRequestMethods.Http.Post;

P a u l 10,766 Reputation points

2021-12-30T00:58:39.92+00:00

Would you be able to share the URL of the download page?
Stacy Bell 41 Reputation points

2021-12-30T01:05:18.353+00:00

I wish I could. It would be a lot easier. Can you give a general snippet to save the file to a location ie desktop?
P a u l 10,766 Reputation points

2021-12-30T01:30:04.057+00:00

You should be able to see what sort of request it is in Chrome if you open the developer tools (F12) then open the Network tab (make sure to enable the Preserve log checkbox). Once you click the download button you should see an entry for the /Monthly Report request and it'll show whether it was GET or POST, and if it was a POST then what the content type of the body was (i.e. application/json, application/x-www-form-urlencoded, etc)

If I had to guess I'd say it was application/x-www-form-urlencoded so this snippet might do what you want (had to pastebin it because MS forums didn't like something in the snippet):

https://pastebin.com/raw/KURyVpz6

This uses a more modern HTTP API in .NET (HttpClient) although it should do the same thing. The values dictionary is the body of the POST request you're sending to the server, which for my example URL above will be ignored, but your server will read, parse & extract the POST fields out of.

It should then copy the response stream of the request to the out.zip file on your desktop (you'll need to update this to match your filename.)
Stacy Bell 41 Reputation points

2021-12-30T16:42:25.507+00:00

I tried your snippet. The file download doesn't contain real data. It has html code.
Bruce (SqlWork.com) 82,221 Reputation points Volunteer Moderator

2021-12-30T17:06:35.273+00:00

The snippet just assumes the names of the post values, they probably not correct. Also you may need to include the button name.

You need to use the browser tools to determine the actual form field names and if it a get or post.

There may also be a required hidden field value and matching cookie value (to prevent xss attacks), that you need to supply. This is a pretty common practice. This will require fetching and parsing the initial page html and cookie, then generating the post request.

Google for posts and tutorials for screen scraping
Stacy Bell 41 Reputation points

2021-12-30T20:29:33.38+00:00

I updated the names of the post values. Where does the button name need to be included? The button is an image inside span tags. I can get the class name in the span tags. The site has JS files I can't access them to see what the class is doing. The download button click request method is a post. The content type is application/json;charset=UTF-8. What should the values be here:
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
P a u l 10,766 Reputation points

2021-12-30T20:58:36.583+00:00
If the request body content-type is application/json then you'll need to use JsonContent instead of FormUrlEncodedContent:

Dictionary<string, string> values = new Dictionary<string, string> { { "start date", "1/1/2021" }, { "end date", "1/15/2021" } }; JsonContent content = JsonContent.Create(values, new MediaTypeHeaderValue("application/json"));

If you can see what request is naturally sent by the app in the Network tab you should just be able to copy the JSON body and populate values in the snippet above to emulate the request. As Bruce mentioned if there are cross-site request forgery tokens also being posted through, or this is an authenticated endpoint that relies on a cookie being passed by the browser this may complicate things. To get a clear idea of what exactly the browser is sending through for the network request you can right click it and copy it was another format that you're familiar with (cURL/Powershell/fetch):

That's the easiest way to emulate a HTTP request exactly in terms of request headers, body, method, etc.

Two things I would check for your HttpClient request are what is the status code? i.e. what value is response.StatusCode - it should be within the success code range.

It sounds like your download page perhaps expects you to be logged in and so is redirecting you to the login page, so I'd expect you'll have a 302 status code (or something in the 3xx range.)
Stacy Bell 41 Reputation points

2021-12-30T22:55:11.907+00:00

The status code is OK. The file download has html code.
P a u l 10,766 Reputation points

2021-12-30T22:58:24.79+00:00

If you copy the HTML, paste it into a .html file and then view it in the browser can you tell what page it is?
Stacy Bell 41 Reputation points

2022-01-01T20:38:32.217+00:00

The html looks like a cloud service that doesn't accept http because it's a secure site, https. I'm reviewing their documentation.

Your answer

P a u l 10,766 Reputation points

2021-12-30T00:13:05.367+00:00

Are you able to access the file in the browser?
Stacy Bell 41 Reputation points

2021-12-30T00:19:18.607+00:00

In the app you need to select a date range then click a download button. I'm trying to do that programmatically. The first line should be this.

var url = string.Format(@"https://www.test.com/Monthly%20Report?start%20date=1/1/2021&end%20date=1/15/2021");
P a u l 10,766 Reputation points

2021-12-30T00:22:03.177+00:00

If you're clicking a button could it be performing a POST request instead of GET?
Stacy Bell 41 Reputation points

2021-12-30T00:53:29.123+00:00

Help

HttpWebRequest httpRequest = (HttpWebRequest) WebRequest.Create("https://www.test.com/Monthly%20Report?start%20date=1/1/2021&end%20date=1/15/2021"); httpRequest.Method = WebRequestMethods.Http.Post;
P a u l 10,766 Reputation points

2021-12-30T00:58:39.92+00:00

Would you be able to share the URL of the download page?
Stacy Bell 41 Reputation points

2021-12-30T01:05:18.353+00:00

I wish I could. It would be a lot easier. Can you give a general snippet to save the file to a location ie desktop?
P a u l 10,766 Reputation points

2021-12-30T01:30:04.057+00:00

You should be able to see what sort of request it is in Chrome if you open the developer tools (F12) then open the Network tab (make sure to enable the Preserve log checkbox). Once you click the download button you should see an entry for the /Monthly Report request and it'll show whether it was GET or POST, and if it was a POST then what the content type of the body was (i.e. application/json, application/x-www-form-urlencoded, etc)

If I had to guess I'd say it was application/x-www-form-urlencoded so this snippet might do what you want (had to pastebin it because MS forums didn't like something in the snippet):

https://pastebin.com/raw/KURyVpz6

This uses a more modern HTTP API in .NET (HttpClient) although it should do the same thing. The values dictionary is the body of the POST request you're sending to the server, which for my example URL above will be ignored, but your server will read, parse & extract the POST fields out of.

It should then copy the response stream of the request to the out.zip file on your desktop (you'll need to update this to match your filename.)
Stacy Bell 41 Reputation points

2021-12-30T16:42:25.507+00:00

I tried your snippet. The file download doesn't contain real data. It has html code.
Bruce (SqlWork.com) 82,221 Reputation points Volunteer Moderator

2021-12-30T17:06:35.273+00:00

The snippet just assumes the names of the post values, they probably not correct. Also you may need to include the button name.

You need to use the browser tools to determine the actual form field names and if it a get or post.

There may also be a required hidden field value and matching cookie value (to prevent xss attacks), that you need to supply. This is a pretty common practice. This will require fetching and parsing the initial page html and cookie, then generating the post request.

Google for posts and tutorials for screen scraping
Stacy Bell 41 Reputation points

2021-12-30T20:29:33.38+00:00

I updated the names of the post values. Where does the button name need to be included? The button is an image inside span tags. I can get the class name in the span tags. The site has JS files I can't access them to see what the class is doing. The download button click request method is a post. The content type is application/json;charset=UTF-8. What should the values be here:
FormUrlEncodedContent content = new FormUrlEncodedContent(values);
P a u l 10,766 Reputation points

2021-12-30T20:58:36.583+00:00

If the request body content-type is application/json then you'll need to use JsonContent instead of FormUrlEncodedContent:

Dictionary<string, string> values = new Dictionary<string, string> { { "start date", "1/1/2021" }, { "end date", "1/15/2021" } }; JsonContent content = JsonContent.Create(values, new MediaTypeHeaderValue("application/json"));

If you can see what request is naturally sent by the app in the Network tab you should just be able to copy the JSON body and populate values in the snippet above to emulate the request. As Bruce mentioned if there are cross-site request forgery tokens also being posted through, or this is an authenticated endpoint that relies on a cookie being passed by the browser this may complicate things. To get a clear idea of what exactly the browser is sending through for the network request you can right click it and copy it was another format that you're familiar with (cURL/Powershell/fetch):

That's the easiest way to emulate a HTTP request exactly in terms of request headers, body, method, etc.

Two things I would check for your HttpClient request are what is the status code? i.e. what value is response.StatusCode - it should be within the success code range.

It sounds like your download page perhaps expects you to be logged in and so is redirecting you to the login page, so I'd expect you'll have a 302 status code (or something in the 3xx range.)
Stacy Bell 41 Reputation points

2021-12-30T22:55:11.907+00:00

The status code is OK. The file download has html code.
P a u l 10,766 Reputation points

2021-12-30T22:58:24.79+00:00

If you copy the HTML, paste it into a .html file and then view it in the browser can you tell what page it is?
Stacy Bell 41 Reputation points

2022-01-01T20:38:32.217+00:00

The html looks like a cloud service that doesn't accept http because it's a secure site, https. I'm reviewing their documentation.

Share via

download file from https with credentials doesn't contain data

Your answer