Invoke-WebRequest and UTF-8

Question

Hi.

I have a function for sending message from powershell to webex by rest api:

Function WebexStringNotify ([String]$string) {
    # $enc = [System.Text.Encoding]::UTF8
    # $dec = [System.Text.Encoding]::Unicode
    # $string = $dec.GetString($enc.GetBytes($string))
    foreach ($roomID in $WebexNotificationRoomIDs) {
        $request = @{
            Uri             = 'https://webexapis.com/v1/messages'
            SessionVariable = 'Session'
            Method          = 'POST'
            Body            = @{
                roomId   = $roomID
                markdown = $string
            }
            Headers         = @{
                               'Authorization' = "Bearer $WebexBotToken"
                               'Accept'        = 'application/json; charset=UTF-8'
                               'Accept-Charset'= 'UTF-8'
                               }
        
        }
        $response = Invoke-WebRequest @request 
        if ($response.StatusCode -eq 200) {
            Write-Host "Webex Notification Sended" -ForegroundColor Green
        }
        else {
            Write-Host "Webex Notification Problem" -ForegroundColor Yellow
            Write-Host $response
        }
    }    
}

Problem is when $string argument contains UTF-8 symbol (ex: "┌") it goes to webex as "┌".

-Console has

$PSDefaultParameterValues['*:Encoding'] = 'utf8'

-Enc/dec:

# $enc = [System.Text.Encoding]::UTF8
# $dec = [System.Text.Encoding]::Unicode
# $string = $dec.GetString($enc.GetBytes($string))

do not work.

-Content-Type header not supported by api, and i suppose it will not work.

-Webex direct input support UTF8 and this char so looks like this is not a webex issue

-Debugging in VS Code shows $string with no problem.

How to resolve this?

Answer 1

This work.

Function WebexStringNotify ([String]$string) {
		$headers = @{
				'Authorization' = "Bearer $token"
				'Accept'        = 'application/json'
		}
		
		$body = @{
                roomId   = $roomId
				markdown = $string
        } | ConvertTo-Json

		$response = Invoke-WebRequest -Uri 'https://webexapis.com/v1/messages' -SessionVariable 'Session' -Method 'POST' -Headers $headers -Body $body -ContentType 'application/json; charset=utf-8'
        Write-Host $response
}

ContentType should not be in Headers for Invoke-WebRequest because it separate argument + converting hashtable body to json

Answer 2

TL;DR

$response = Invoke-WebRequest @request 
$content = [System.Text.Encoding]::UTF8.GetString($response.Content.ToCharArray())
Write-Host $content

Explanation

The problem is that the resulting string of $response.Content gets automatically interpreted as a Unicode (typically UTF-16LE) string. Therefore [System.Text.Encoding]::Unicode.GetBytes() won't work since it's operating on a string which has already been (faulty) interpreted as Unicode. So it will convert the faulty string symbols to a byte array which is not what you want. Instead you want to handle the original Content without prior interpretation. Therefore ToCharArray() is used to interpret the incoming response as a byte array. About this we know that it encodes a UTF-8 string so we can use UTF8.GetString() to retrieve a correct string object.

Answer 3

This sends the message body as UTF8. The character in the "$string" variable is represented in UTF8 as a sequence of three hex values: 0xE2 0x94 0x8C

This is what the message looks like (captured by Fiddler):

POST http://192.168.1.202/ HTTP/1.1
Accept: application/json
Content-Type: application/json; charset=utf-8
User-Agent: Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.19041.2673
Host: 192.168.1.202
Content-Length: 55
Connection: Keep-Alive

{
    'roomId"    = 'usa',
    'markdown'  = '┌'
}

Here's what that message looks like in hex. The three highlighted characters (in hex) is the character '┌'.

Here's the code:

$string = [char]0x250c # "┌" Unicode U+250C, UTF8 0xE2 0x94 0x8C 

# body is JSON, not a PowerShell hash
$body = @"
{
    'roomId"    = 'usa',
    'markdown'  = '$string'
}
"@

$request = @{
    Uri             = 'http://192.168.1.202'
    SessionVariable = 'Session'
    Method          = 'POST'
    Body            = [System.Text.Encoding]::UTF8.GetBytes($body)
    Headers         = @{"Content-Type" = "application/json; charset=utf-8"
                       'Accept'        = 'application/json'
                       }

}

$response = Invoke-WebRequest @request