Problem with string encoding from Web when showed in VB 2015 or VB 2022 Application.

Question

Hi,

Simple question :
I use Reverse Geocoding to get info from a GPS Point.
I get a Response string ( XML or JSON ) which is always correct in every browser I use.

When I use VB Application to build Request String I get some strange Chars in Response String,

always related to City Names, Streets and so on.

Please notice that Web gives right place names.
The problem is how VB gets the XML / JSON Document...

For Example :
Point : 41.63077140326721, -4.5227111817070345
GeoCoding finds correctly a town in Spain : Villabáñez
In Browser the Xml/Json Document returned shows correctly that name.
In VB Application if I see the document text it has changed in : **Villabáñez
**
I use this code to get the response :

        Using WC As New System.Net.WebClient
            Try
                strResponse = WC.DownloadString(strRequest)
                XMLDoc.LoadXml(strResponse)
            Catch ex As Exception
                'Never Got Errors ...
            End Try
        End Using

I think some encoding work has to be done to put things allright,

but I don't know how to fix it.

Thanks to anyone able to help.

Answer 1

Thanks for reaching out with this issue regarding string encoding when retrieving data from web in you VB application. The issue you are experiencing where characters like the accepted ‘á’ in ‘Villabáñez’ appear corrupted in your VB application but display correctly in a web browser is almost certainly an encoding mismatch.

When you use WC.DownloadString(strRequest), the System.Net.WebClient attempts to determine the character encoding of the response. By default , if the response headers don’t specify an encoding (like charset = utf – 8), WebClient often defaults to an encoding like ISO-8859-1 or a system default encoding (like Windows-1252 or ASCII). However, modern web services, especially those dealing with Spanish place names are most certainly sending the data encoded in UTF-8.

When WebClient interprets UTF-8 bytes as if the were ISO-8859-1, multi-byte characters like ‘á’ (which is two bytes in UTF-8) are incorrectly represented as two separate characters from the ISO-8859-1 set, resulting in the “mojibake” you see.

Solution: specify UTF-8 Encoding

The fix is to explicitly tell the WebClient to use UTF-8 encoding for the download string

Instead of using WC.DownloadString(strRequest), you should use WC.DownloadData to get raw bytes and then use the Encoding.UTF8 class to convert those bytes into a string.

Here’s the updated code of VB:

         Using WC As New System.Net.WebClient

         Try

                1.      Download the raw data (bytes) instead of a string Dim byteResponse() As Byte = WC.DownloadData(strRequest)

                2.      Explicitly convert the bytes to a string using UTF-8 encoding Dim strResponse As String= System.Text.Encoding.UTF8.GetString(byteResponse)
            Now strResponse should contain ‘Villabáñez’ correctly                 

            Continue with your existing logic XMLDoc.LoadXml(strResponse)          

Catch ex As Exception            

            ·         Never Got Errors …              

            ·         Note: use a real error log or message box here in a production environment     End Try       

End Using

 

Why this works:

1.      DownloadData: this method fetches the response content as a raw array of bytes (byteResponse()). This prevents the WebClient from attempting an incorrect default conversion.

2.      System.Text.Encoding.UTF8.GetString(byteResponse): This is the critical step. It correctly decodes the sequence of the bytes using the UTF-8 standard, which preserves the special characters as intended by geocoding service.
Let me know if you need any further help with this. We'll be happy to assist.

If you find this helpful, please mark this as answered.

Answer 2

Thanks for attention and explanation.

I had already found the solution with :

WC.Encoding = System.Text.Encoding.UTF8

        Using WC As New System.Net.WebClient
            WC.Encoding = System.Text.Encoding.UTF8
            Try
                'WC.Headers("User-Agent") = _BrowserUserAgent
                strResponse = WC.DownloadString(strRequest)
                ...
                ...
            Catch ex As Exception
                ...
            End Try
        End Using

Instead of trying to find the right encoding on the response string returned by web, the solution is to set the UTF8 Encoding on the WebClient Object before the request operation.

Your solution works too, so I can mark it as answer. Thanks again.