C++: Reading an UTF-8 file with Win32 API

Question

C++: Reading an UTF-8 file with Win32 API

thebluetropics 1,046

I have entering the file IO section of Win32 API. This time, I wanted to read file from.

Here is what I tried so far. This code outputs ￣ﾃﾨ￣ﾃﾫ￣ﾂﾷ￣ﾂﾫ instead of Japanese word ヨルシカ which is weird.
I know this problem, in this code, I didn't read the file as an UTF-8 file, instead I read the file as an ANSI file (256 characters).

So how can I read an UTF-8 file and then put the result in wchar_t array, using built-in Win32 API?

I want the readed character as single UTF-8 character. So each character in the buffer is UTF-8 character, not 256 character.

Igor Tandetnik 1,116 Reputation points

2022-09-27T22:13:26.903+00:00

std::wstring wstr(str.begin(), str.end()); doesn't convert UTF-8 to Unicode. It converts each individual byte (octet) of the narrow string to the UTF-16 codepoint with the same numerical value. All characters in the resulting wstr are in U+0000 to U+00FF range.

If you insist on Win32 API alone, and not the C++ standard library or any third-party library, then MultiByteToWideChar is the only way to make this conversion, but it doesn't deal with the buffer ending mid-character. At this point, your best bet is probably to implement your own; conversion between UTF-8 and UTF-16 is straightforward, once you understand the details of both encodings.
thebluetropics 1,046 Reputation points

2022-09-28T08:29:05.043+00:00

I think the same. Probably the only way to do this is by implement my own Multibyte to UTF-8 conversion. However, what about other popular applications (e.g Notepad++)? I don't think they implement such thing for their own "file to display" converter. I will stick to the MultiByteToWideChar() for now, as it is the only solution available within Win32 API.

Answer accepted by question author

1 additional answer

Your answer

Igor Tandetnik 1,116 Reputation points

2022-09-27T22:13:26.903+00:00

std::wstring wstr(str.begin(), str.end()); doesn't convert UTF-8 to Unicode. It converts each individual byte (octet) of the narrow string to the UTF-16 codepoint with the same numerical value. All characters in the resulting wstr are in U+0000 to U+00FF range.

If you insist on Win32 API alone, and not the C++ standard library or any third-party library, then MultiByteToWideChar is the only way to make this conversion, but it doesn't deal with the buffer ending mid-character. At this point, your best bet is probably to implement your own; conversion between UTF-8 and UTF-16 is straightforward, once you understand the details of both encodings.
thebluetropics 1,046 Reputation points

2022-09-28T08:29:05.043+00:00

I think the same. Probably the only way to do this is by implement my own Multibyte to UTF-8 conversion. However, what about other popular applications (e.g Notepad++)? I don't think they implement such thing for their own "file to display" converter. I will stick to the MultiByteToWideChar() for now, as it is the only solution available within Win32 API.

Answer 1

Jeanine Zhang-MSFT 11,356 Microsoft External Staff

Hi，@thebluetropics

As far as I'm concerned you could try to use MultiByteToWideChar function with CP_UTF8.

thebluetropics 1,046 Reputation points

2022-09-27T08:05:13.437+00:00

@Jeanine Zhang-MSFT

I have to think about reading the file chunk-by-chunk. Each ReadFile() can read up to 4 bytes, when the MultiByteToWideChar() read the end of the buffer, it has a probability of incorrect decoding due to separated code units between chunks.

The probability of incorrect decoding is high in this case.

Answer 2

Guido Franzke 2,191

Hello,
have a look at this: read-unicode-utf-8-file-into-wstring
Regards, Guido

thebluetropics 1,046 Reputation points

2022-09-27T06:15:53.903+00:00

Hi, @Guido Franzke

Thank you for the response.

The stackoverflow question you refer is what I read, and it is does not uses Win32 API. It's uses C++ standard wifstream.
Guido Franzke 2,191 Reputation points

2022-09-27T07:22:54.477+00:00

You can use Standard-C++ in a Win32 C++ app.

Share via

C++: Reading an UTF-8 file with Win32 API

1 additional answer

Your answer