How to get the (full) content of a webpage by Powershell

Question

How to get the (full) content of a webpage by Powershell

Eddy De Maerschalck 0

I'm trying to look at the content of a dynamically written webpage (by a supplier). As it takes sometimes for that supplier a long time to get all the data and write the webpage, my powershell instructions to consult that page doesn't have all the data.

How can I use the instruction invoke-webrequest to wait long enough to get all the data.

For example

$from = "2023-07-10"

$until = "2023-07-17"

$url = "https://www.europarcs.nl/zoek/verblijven?resort=DP800&startdate=" + $from + "&enddate=" + $until + "&subjects=114106,2"

$url

$r = invoke-webrequest -uri $url

$r.content.contains('Cube')

gives me now the result false even when i see visually the word Cube on the screen when i look at the page https://www.europarcs.nl/zoek/verblijven?resort=DP800&startdate=2023-07-10&enddate=2023-07-17&subjects=114106,2 in google chrome webbrowser

(Attention, $from en $until has to be in the future (and mostly the best result is when you use a monday and a friday).

I have also experiment with the InternetExplorer.Application object, but there i had also the same problem. I'v use the properties busy and readystate to build in some delay, but the same problem.

Thanks in advance for any idea's.

Eddy

Eddy De Maerschalck 0

Another experiment with

$webClient = New-Object System.Net.WebClient $content = $webClient.DownloadString($url)

gives the same problem!

$url = 'https://www.europarcs.nl/zoek/verblijven?resort=DP800&startdate=2023-7-3&enddate=2023-7-7&subjects=114106,2'
$webClient = New-Object System.Net.WebClient
$content = $webClient.DownloadString($url)

if ($content -like '*Cube*') {
    Write-Host "The word 'Cube' is present in the webpage content."
} else {
    Write-Host "The word 'Cube' is not found in the webpage content."
}

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

1 answer

Your answer

Eddy De Maerschalck 0 Reputation points

2023-06-21T14:18:10.5033333+00:00

Another experiment with

$webClient = New-Object System.Net.WebClient $content = $webClient.DownloadString($url)

gives the same problem!

$url = 'https://www.europarcs.nl/zoek/verblijven?resort=DP800&startdate=2023-7-3&enddate=2023-7-7&subjects=114106,2' $webClient = New-Object System.Net.WebClient $content = $webClient.DownloadString($url) if ($content -like '*Cube*') { Write-Host "The word 'Cube' is present in the webpage content." } else { Write-Host "The word 'Cube' is not found in the webpage content." }
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Answer 1

Rich Matheisen 47,906

The content you want doesn't exist in the data you received from the URL. What you see is generated by scripts within that data that run to produce the web page you see.

Eddy De Maerschalck 0 Reputation points

2023-06-21T21:21:48.2466667+00:00

Any idea how to capture the result of that script?

Any idea how to capture the result of that script?

How to determine which script?
Rich Matheisen 47,906 Reputation points

2023-06-22T01:59:02.94+00:00

The contents are just one large script. Even if you save the web page as a html file you'll only get the script, not the images it retrieves.

OCR might be a way to accomplish what you want.

Share via

How to get the (full) content of a webpage by Powershell

1 answer

Your answer