Powershell Invoke-WebRequest not timing out

Question

I've created a script that downloads files from various sources and save these to a local network share for processing

If I run the script directly from powershell it works and reports error if cannot find the file needed

But if I run the powershell script through SQL Server, if the web file is not found it just hangs the job and does not timeout or report back the error 404

using the command

$Response = Invoke-WebRequest -Uri $source -Outfile $destination -Credential $cred
$StatusCode = $Response.StatusCode

I've tried using the -TimeoutSec and -MaximumRetryCount but these don't work either

Any ideas

Thanks

Answer 1

Found the issue, its when it can't overwrite the file during the download process it fails and just hangs

Answer 2

ignore the final errors being returned back to sql server agent, this is not the problem

i already use try/catch but it doesn't get to the catch when the url is not found which is why i'm confused

i have it downloading multiple files and if any fails it sets the marker to say the process has failed but via sql server agent it is not doing this step correctly as it just hangs, but if i run the same script via powershell ise the catch works and returns the 404 which i then set flag to say the process has failed.

ie

try {
$Response = Invoke-WebRequest -Uri "https://mywebsite.com/myfiletodownload.zip"-Outfile "c:\mydownloads\downloadedfile.zip" -Credential $cred
$StatusCode = $Response.StatusCode
}
catch
{
$StatusCode = $_.Exception.Response.StatusCode.value__
}

Answer 3

i have it downloading multiple files

Build a test script that all it does is download one file that does not exist. That way you can verify that your error checking works and other code that you have in the script isn't causing the hang.

but if i run the same script via powershell ise the catch works

The issue with ISE is that it doesn't reset all variables when you hit F5 to run the script. So it you run the script multiple times, you may be referencing variables that were set during a prior execution. Are you running ISE on the server or on your desktop?

In the testing that I did in PS 5.1, this statement...

$StatusCode = $.Exception.Response.StatusCode.value_

...should be

$StatusCode = $_.Exception.Response.StatusCode

Also add a transcript. Review the contents of the 2 log files after you have SQL run it.

This should be the entire script.

    start-transcript -Path c:\temp\ps.log
    try {
        # add your code to build $cred here

        "************** Downloading the file **************" | out-file c:\temp\MyScript.log
        $Response = Invoke-WebRequest -Uri "https://www.microsoft.com/myfiletodownload.zip" -Outfile "c:\temp\downloadedfile.zip" -Credential $cred
        "Invoke-webrequest has finished" | out-file c:\temp\MyScript.log -Append
        $StatusCode = $Response.StatusCode
    }
    catch
    {
        "************** We crashed ******************" | out-file c:\temp\MyScript.log -Append 
        $_.exception.message | out-file c:\temp\MyScript.log -Append
        $StatusCode = $_.Exception.Response.StatusCode
    }
    "************** Complete **********************" | out-file c:\temp\MyScript.log -Append
    "Statuscode = {0}" -f $StatusCode | out-file c:\temp\MyScript.log -Append
    Stop-Transcript

Assuming that works, then test a successful download.

Answer 4

Can you identify the URLs that cause the problem? If so, do those URLs cause the same problem when run from PowerShell?

I've had problems (hanging) similar to this due to problems with the HTML returned from a site (including Microsoft). The problem isn't with PowerShell but with the HTML parser in the O/S.

Try adding "-UseBasicParsing" to the Invoke-WebRequest. The HTML code will still be still available in the Content property of your $Response object.