Get data from webpage using ElementId's and WebView2

Question

I used WebBrower for a few years successfully getting data from a webpages elements. Now WebBrower doesn't support the webpage anymore, so I use WebView2. After reading the documentation it appears I have to execute JavaScript asynchronously to get the data. However, I'm having trouble applying the examples to my requirement.

Below is some sample code.

1) I want to use an elementId to to test if it exists, if exist then get the elements value, and click it, in that order, but my code won't compile, why?

2) In my Button1_Click event, I have to call DoEvents to force WebView2 to navigate, otherwise it won't navigate to the url until the click event is done executing. How can I force WebView2 to navigate?

 Imports System.Threading.Tasks
    Imports Microsoft.Web.WebView2.Core

    Public Class Form1
        Private url As String = "https://www.microsoft.com"

        Public Sub New()
            InitializeComponent()
            Me.InitializeCoreWebView2()   ' this must be here so that Core is initalized when form is loaded
        End Sub

        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles RadButton1.Click
            Me.WebView21.CoreWebView2.Navigate(url)

            Application.DoEvents()

            ' do more stuff
        End Sub

        Private Async Sub InitializeCoreWebView2()
            'LogFileWrite("Await WebView21.EnsureCoreWebView2Async")
            Await Me.WebView21.EnsureCoreWebView2Async(Nothing)
        End Sub

        Private Sub WebView21_NavigationCompleted(sender As Object, e As CoreWebView2NavigationCompletedEventArgs) Handles WebView21.NavigationCompleted
            If e.IsSuccess Then
                If Me.WebView21.CoreWebView2.Source = url Then
                    ' get "Microsoft 365" text
                    Dim jScript As String = "document.getElementById('shellmenu_0').value;"
                    Dim t As Task(Of String) = Await Me.WebView21.CoreWebView2.ExecuteScriptAsync(jScript)        '<--- won't compile, why?
                    Debug.Print(t.Result)

                    ' click the "Microsoft 365" tab
                    jScript = "document.getElementById('shellmenu_0').click();"
                    Await Me.WebView21.CoreWebView2.ExecuteScriptAsync(jScript)        '<--- won't compile, why?
                End If
            Else
                ' LOG ERROR HERE
            End If
        End Sub
    End Class

Thank you

Answer 1

You don't need to call DoEvents or even EnsureCoreWebView2Async
There are several ways to initialize WebView2
The simplest one is to call :

WebView21.Source = New Uri("https://www.google.com")

then you must add Async at :

Private Async Sub WebView21_NavigationCompleted

and you can execute scripts, like :

Dim sHTML As String = Await WebView21.ExecuteScriptAsync("document.documentElement.outerHTML;")

Answer 2

Hi @Ryan Hennings ,

1) I want to use an elementId to to test if it exists, if exist then get the elements value, and click it, in that order, but my code won't compile, why?

The method in which Await is used must have an Async modifier. Await Operator (Visual Basic)

2) In my Button1_Click event, I have to call DoEvents to force WebView2 to navigate, otherwise it won't navigate to the url until the click event is done executing. How can I force WebView2 to navigate?

I tried to delete the DoEvents in your code, it completed the navigation before the click event was executed.
So DoEvents() is not necessary
Best Regards.
Jiachen Li

---

If the answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 3

Thank you for the replies! This partially helped me. However, WebView2 will not navigate to the URL until the click code is done executing. I need to scrap the page that I'm trying to navigate to, but the page doesn't load until after the click event. DoEvents sort of helps, but it is inconsistent. Any idea how to force WebView2 to load before proceeding?

          Private Sub Button1_Click(sender As Object, e As EventArgs) Handles RadButton1.Click
              Me.WebView21.CoreWebView2.Navigate(url)
              Application.DoEvents()
              ' wait for page to load, then do more stuff
          End Sub

Answer 4

Hi @Ryan Hennings ,
You can set a boolean value to let the code continue execution after the navigation of WebView2 is complete.
Here is my test code which you can refer to.

Public Class Form1  
    Private url As String = "https://www.microsoft.com"  
    Dim fns As Boolean = False  
    Public Sub New()  
        InitializeComponent()  
    End Sub  
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click  
        WebView21.Source = New Uri(url)  
        fns = False  
        Do Until fns  
            Application.DoEvents()  
        Loop  
        Console.WriteLine("1")  
    End Sub  
    Private Sub WebView21_NavigationCompleted(sender As Object, e As CoreWebView2NavigationCompletedEventArgs) Handles WebView21.NavigationCompleted  
        fns = True  
        Console.WriteLine("2")  
        If e.IsSuccess Then  
            If Me.WebView21.CoreWebView2.Source = url Then  
                ' get "Microsoft 365" text  
                Dim jScript As String = "document.getElementById('shellmenu_0').value;"  
                Dim t As Task(Of String) = Me.WebView21.CoreWebView2.ExecuteScriptAsync(jScript)  
                Debug.Print(t.Result)  
                ' click the "Microsoft 365" tab  
                jScript = "document.getElementById('shellmenu_0').click();"  
                Me.WebView21.CoreWebView2.ExecuteScriptAsync(jScript)  
            End If  
        Else  
            ' LOG ERROR HERE  
        End If  
    End Sub  
End Class  

Hope the code above could be helpful.
Best Regards.
Jiachen Li

----------

If the answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 5

Here's an updated snippet of my project. Place a Button and WebView2 control on a Form and test code below.

I need to load multiple URL's and "scrape" data from each URL one at a time. Unfortunately, WebView2 requires the URL to be loaded from the UI thread, so I can't use threading. This is problematic because the UI won't load the URL until the Button1_Click event is done executing. As a workaround, I tried Application.DoEvents, but my code crashes when processing the 3rd URL, why?

I'm confused and need help. I want to force WebView2 to fully load a URL, run a series of functions to get the data from the page, save to database, and continue to process the next URL. How do I do that?

Imports System.Threading.Tasks
Imports Microsoft.Web.WebView2.Core

Public Class Form1

    Private _isNavigating As Boolean

    Public Sub New()
        InitializeComponent()
        Me.InitializeAsync()
    End Sub

    Private Async Sub InitializeAsync()
        Await Me.WebView21.EnsureCoreWebView2Async
    End Sub

    Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

        Dim uris As New List(Of Uri)
        uris.Add(New Uri("https://qpublic.schneidercorp.com/Application.aspx?AppID=794&PageTypeID=4&KeyValue=C035000070032000"))
        uris.Add(New Uri("https://qpublic.schneidercorp.com/Application.aspx?AppID=794&PageTypeID=4&KeyValue=0049B00000026000"))
        uris.Add(New Uri("https://qpublic.schneidercorp.com/Application.aspx?AppID=794&PageTypeID=4&KeyValue=X006000000030000"))
        uris.Add(New Uri("https://qpublic.schneidercorp.com/Application.aspx?AppID=794&PageTypeID=4&KeyValue=8000000371000"))

        For Each uri As Uri In uris
            Me.WebView21.CoreWebView2.Navigate(uri.ToString)

            ' force WebView2 to load webpage before continuing
            Me._isNavigating = True
            Do
                Application.DoEvents()
            Loop While Me._isNavigating
        Next
    End Sub

    Private Async Function GetAPN(webView As WebView2) As Task(Of String)
        Dim apn As String = Await webView.ExecuteScriptAsync("document.querySelector('#ctlBodyPane_ctl00_ctl01_lblParcelID').innerText;")
        If apn = "null" Then
            Throw New Exception("APN not found in HTMLDocument")
        Else
            apn = Trim(apn.Replace("""", ""))
            Return apn
        End If
    End Function

    Private Async Function GetAddress(webView As WebView2) As Task(Of String)
        Dim address As String = Await webView.ExecuteScriptAsync("document.querySelector('#ctlBodyPane_ctl00_ctl01_lblLocationAddress').innerText;")
        If address = "null" Then
            Throw New Exception("Address not found in HTMLDocument")
        Else
            address = Trim(address.Replace("""", ""))
            Return address
        End If
    End Function

    Private Sub WebView21_NavigationStarting(sender As Object, e As CoreWebView2NavigationStartingEventArgs) Handles WebView21.NavigationStarting
        Debug.Print("NavigationStarting")
    End Sub

    Private Async Sub WebView21_NavigationCompleted(sender As Object, e As CoreWebView2NavigationCompletedEventArgs) Handles WebView21.NavigationCompleted
        Debug.Print("NavigationCompleted")
        Me._isNavigating = False

        ' get data from multiple elements on loaded webpage
        Debug.Print("APN:  " & Await Me.GetAPN(Me.WebView21))
        Debug.Print("Address:  " & Await Me.GetAddress(Me.WebView21))
    End Sub

End Class