VB.NET Winform HttpWebRequest / XmlDocument Error: The remote server returned an error: (403) Forbidden.

Question

So the below code works on my home network but at work .NET application gets a 403 Forbidden. I believe ZScaler firewall is blocking traffic, but I can view XML via browser without issue just not in a .NET application. I have read that it's trying to block BOT traffic so adding a .UserAgent should fix the issue, It did not.

I am just trying to read an XLM file from the internet and can view from a webrowser.

What alternatives are there for this issue?

Try
            Dim url As String = "https://www.espn.com/espn/rss/news"
            Dim rssFeed As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)

            'rssFeed.Method = "POST"
            'rssFeed.Accept = "application/xml"
            'rssFeed.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
            'rssFeed.Credentials = CredentialCache.DefaultNetworkCredentials
            'rssFeed.Proxy.Credentials = CredentialCache.DefaultCredentials

            Dim response = rssFeed.GetResponse()
            Dim rssStream = response.GetResponseStream()

            Dim rssDoc As New XmlDocument()
            rssDoc.Load(rssStream)
            Dim rssItems As XmlNodeList = rssDoc.SelectNodes("rss/channel/item")
            Dim i As Integer = 0
            Dim dt As DataTable = New DataTable("table")
            dt.Columns.Add("Title", Type.GetType("System.String"))
            dt.Columns.Add("Description", Type.GetType("System.String"))
            dt.Columns.Add("Link", Type.GetType("System.String"))

            While i < rssItems.Count
                Dim node As XmlNode = rssItems.Item(i).SelectSingleNode("title")
                Dim title As String
                Dim description As String
                Dim link As String
                If node IsNot Nothing Then
                    title = node.InnerText
                Else
                    title = ""
                End If

                node = rssItems.Item(i).SelectSingleNode("description")
                If node IsNot Nothing Then
                    description = node.InnerText
                Else
                    description = ""
                End If

                node = rssItems.Item(i).SelectSingleNode("link")
                If node IsNot Nothing Then
                    link = node.InnerText
                Else
                    link = ""
                End If
                Dim dr As DataRow = dt.NewRow()
                dr("Title") = title
                dr("Description") = description
                dr("Link") = link

                dt.Rows.Add(dr)

                i += 1
            End While

            DataGridView1.DataSource = dt
            WebBrowser1.ScriptErrorsSuppressed() = True
            WebBrowser1.Navigate("https://www.espn.com/espn/rss/news")

        Catch ex As Exception
            MsgBox(ex.Message)
        End Try

Answer 1

Hi. Thank you for your question and reaching out. I’d be more than happy to help you with your query.

The error message "The remote server returned an error: (403) Forbidden" when using HttpWebRequest and XmlDocument in a VB.NET Winform application typically indicates that the server is refusing the request due to a lack of proper authentication or authorization. Here are a few possible causes and solutions:

1. Incorrect credentials: Verify that the credentials being used to authenticate the request are correct and have the necessary permissions to access the resource.
2. Expired or invalid token: If you are using token-based authentication, verify that the token is still valid and has not expired.
3. Firewall or proxy blocking: Verify that the request is not being blocked by a firewall or proxy server.
4. Not authorized to access the resource: Verify that the user has the necessary permissions to access the resource.
5. Missing headers: Verify that the necessary headers are being sent with the request, such as the Authorization header or the Content-Type header.

6.Missing or incorrect endpoint URL: Verify that the endpoint URL is correct and that the resource is accessible.

7. Missing or incorrect method: Verify that the correct method is being used for the request (GET, POST, etc.)

It is a good idea to check the documentation of the API or service that you are trying to access for more information on the required authentication and authorization methods, as well as any specific requirements for the request headers.

If the reply was helpful, please don’t forget to upvote or accept as answer, thank you.