Invoke-webrequest downloads a few kb and stops

IT Magician 1 Reputation point
2022-10-06T01:01:34.637+00:00

Hi Community,

I have an issue I can re-create and need guidance. I can successfully do a Invoice-Webrequest for tiny files works without issue. With Powershell 5 when I try to pull down a 41MB file it downloads 16 to 22kb and stops. No errors, just stops.

In Powershell 7 when I try to pull down the 41MB file it stops and doesn't do anything.

This seems like a very simply command, so I am trying to identify what I am doing wrong. If I browse the URL through a web browser it downloads all 41MB without issue.

Windows Server PowerShell
Windows Server PowerShell
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.PowerShell: A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
5,383 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Rich Matheisen 45,096 Reputation points
    2022-10-06T02:05:28.827+00:00

    Try this and see if it completes: 

    $uri = "https://my.url.here.org"  
$client = New-Object System.Net.WebClient  
$client.DownloadFile($uri, 'file-path-to-save-file-to.html');

    NOTE: this quick example does absolutely no error checking!

    If that works, try adding the "-UseBasicParsing" switch to the Invoke-WebRequest cmdlet.

    0 comments
  2. Limitless Technology 39,381 Reputation points
    2022-10-07T14:38:34.79+00:00

    Hello,

    In order to download a file, Invoke-WebRequest isn't the most optimal way as the HTTP response stream is buffered into memory, and once the file has been fully loaded- then only it will be flushed to disk. This can cause a performance impact in case of large files.

    I would suggest you to use the System.Net.WebClient DotNET class to download files from your GitHub source. You can refactor your code to something like this:

    $url = "<URLpath>"
    $output = "C:\SomePath\filename"
    (New-Object System.Net.WebClient).DownloadFile($url, $output)

    How is this cmdlet better than Invoke-WebRequest? You might ask.

    With System.Net.WebClient, the speed/performance gets improved a lot as the HTTP response stream is buffered directly to disk throughout the download process (and not splitting the work into fetch-and-flush tasks).

    Note: Make sure the local output file (for which you're providing the path in $output) is a valid file and it exists, or else you might get some error while using DownloadFile method.

    Additionally since the above solution doesn't seem to be working as expected in case of compressed files, here's another workaround that can be used for achieving this using PowerShell:

    $url = "<URLpath>"
    $zipOutput = "C:\Output\" + $(Split-Path -Path $url -Leaf)
    $extractedOutput = "C:\ExtractedOutput\"
    (New-Object System.Net.WebClient).DownloadFile($url, $zipOutput)
    $shellObj = New-Object -ComObject Shell.Application
    $files = $shellObj.Namespace($zipOutput).Items()
    $shellObj.NameSpace($extractedOutput).CopyHere($files)
    Start-Process $extractedOutput

    The zip file will be downloaded to the path provided in $zipOutput, and the script will further extract the contents & store extracted contents in the path provided in $extractedOutput. Make sure that the 'C:\ZipOutput' and 'C:\ExtractedOutput' folders exist on your machine where you're executing this script.

    --If the reply is helpful, please Upvote and Accept as answer--