Web Page Question

Question

Web Page Question

Devon Nullman 21

I want to grab the available links on a webpage that displays search results. I am using System.Net.WebRequest to load the data then I parse it.

The problem is that the page only shows 24 hits and if there are more than 24, you must scroll to the bottom to get another 6 or displayed.

Is there any way to simulate the scrolling ? If I had to, I could load the page into a Web Browser.

1 answer

Your answer

Answer 1

Jiachen Li-MSFT 34,231 Microsoft External Staff

Hi @Devon Nullman ,
If you load the page into a Web Browser, you can use the code below to grab the available links.

    Private Sub WebBrowser1_DocumentCompleted(sender As Object, e As WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted  
        For Each ClientControl As HtmlElement In WebBrowser1.Document.Links  
            Dim href As String = ClientControl.GetAttribute("href").ToString  
            If href.Contains("https://") Then  
                ListBox1.Items.Add(href)  
            End If  
        Next  
    End Sub

Hope the code above could be helpful.
Best Regards.
Jiachen Li
*
If the answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Devon Nullman 21 Reputation points

2021-11-09T16:34:04.48+00:00

Thanks, but that code gets the links but only the first page.

Jiachen Li-MSFT 34,231 Microsoft External Staff

Hi @Devon Nullman ,

I do some tests to auto click the next-page element in order to get multi-page results.
Note that different search engines have different next-page element characteristics.
The following code implements the effect of automatic page turning to get the next page.

    Dim page As Integer = 10  
    Private Sub WebBrowser1_DocumentCompleted(sender As Object, e As WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted  
        For Each ClientControl As HtmlElement In WebBrowser1.Document.Links  
            Dim href As String = ClientControl.GetAttribute("href").ToString  
            If href.Contains("https://") Then  
                ListBox1.Items.Add(href)  
            End If  
            'Bing'  
            If href.Contains("FORM=PORE") Then  
                ClientControl.InvokeMember("click")  
            End If  
            'Google'  
            If href.Contains($"&start={page}&sa=N") Then  
                page = page + 10  
                ClientControl.InvokeMember("click")  
            End If  
        Next  
    End Sub

Jiachen Li-MSFT 34,231 Reputation points Microsoft External Staff

2021-11-23T09:40:14.51+00:00

Hi @Devon Nullman ,
May I know if you have got any chance to check my answer? I am glad to help if you have any other questions.

Share via

Web Page Question

1 answer

Your answer