How to set text encoding on a data column from EntityFramework or ADO.NET?

Question

How to set text encoding on a data column from EntityFramework or ADO.NET?

Juan Dent 226

I have a native C++ Windows MFC application that accesses sqlite3.dll to load data from a database and it loads non-ASCII characters like 'ñ' without changing them to '?'
When I access this database from EntityFramework, the non-ASCII characters get represented like '?' instead. Where can I configure EF or ADO.NET to correct this?

Jack J Jun 21,576 Reputation points Microsoft Vendor

2022-08-31T02:14:54.337+00:00

@Juan Dent , Welcome to Microsoft Q&A, Could you tell me how do you define the related field in your database?

Juan Dent 226

CREATE TABLE Doctors (  
    id_doctor      INTEGER PRIMARY KEY AUTOINCREMENT  
                           NOT NULL,  
    first_name     TEXT    COLLATE NOCASE  
                           NOT NULL,  
    last_name      TEXT    NOT NULL,  
    fkey_specialty INTEGER NOT NULL,  
    FOREIGN KEY (  
        fkey_specialty  
    )  
    REFERENCES Specialties (id_specialty)   
);

The related field is either first_name or last_name., both behave the same when dealing with non-ASCII characters

Thanks

AgaveJoe 19,046 Reputation points

2022-08-31T17:56:52.147+00:00

I cannot reproduce this issue with default installation of Sqlite an ASP.NET Core MVC application.

I'm guessing there is something else going on with your application. How or where are you displaying the text?
Juan Dent 226 Reputation points

2022-08-31T20:54:57.067+00:00
I am writing spanish text in a MFC EditBox and reading its value into a CString. The letter ñ inside the CString has a value of 241. Then I convert from CString (which is a wide string) to 7 bit string by using this code:

// convert wstring to string inline std::string to_string(const std::wstring& str, const std::locale& loc = std::locale{}) { std::vector<char> buf(str.size()); std::use_facet<std::ctype<wchar_t>>(loc).narrow(str.data(), str.data() + str.size(), '?', buf.data()); return std::string(buf.data(), buf.size()); }

And when I inspect the returned string the value for the ñ is -15 (which is 241-256) -- its out of range for a 8 bit signed char (which is 7 bits + the sign bit)...

Inside the debugger ñ is well displayed as ñ. All seems to be okay but when I load the string in a .NET application, the character ñ value is 65533 and it does not display as ñ!!

So it is an encoding problem!!
Juan Dent 226 Reputation points

2022-08-31T20:59:46.803+00:00

Looking at the character map application that is part of Windows 10 Pro, I can verify that ñ has a value of 241 in the unicode character set as well as the DOS : United States character set which is also 241
Laxmikant 121 Reputation points

2022-09-04T02:39:56.233+00:00
you can try

Convert.ToBase64String(Encoding.UTF8.GetBytes(dirtyString)))

https://geeksarray.com/blog/using-fluent-api-in-efcore-code-first

Jack J Jun 21,576 Reputation points Microsoft Vendor

2022-08-31T02:14:54.337+00:00

@Juan Dent , Welcome to Microsoft Q&A, Could you tell me how do you define the related field in your database?
Juan Dent 226 Reputation points

2022-08-31T16:27:19.907+00:00

CREATE TABLE Doctors ( id_doctor INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, first_name TEXT COLLATE NOCASE NOT NULL, last_name TEXT NOT NULL, fkey_specialty INTEGER NOT NULL, FOREIGN KEY ( fkey_specialty ) REFERENCES Specialties (id_specialty) );

The related field is either first_name or last_name., both behave the same when dealing with non-ASCII characters

Thanks
AgaveJoe 19,046 Reputation points

2022-08-31T17:56:52.147+00:00

I cannot reproduce this issue with default installation of Sqlite an ASP.NET Core MVC application.

I'm guessing there is something else going on with your application. How or where are you displaying the text?
Juan Dent 226 Reputation points

2022-08-31T20:54:57.067+00:00

I am writing spanish text in a MFC EditBox and reading its value into a CString. The letter ñ inside the CString has a value of 241. Then I convert from CString (which is a wide string) to 7 bit string by using this code:

// convert wstring to string inline std::string to_string(const std::wstring& str, const std::locale& loc = std::locale{}) { std::vector<char> buf(str.size()); std::use_facet<std::ctype<wchar_t>>(loc).narrow(str.data(), str.data() + str.size(), '?', buf.data()); return std::string(buf.data(), buf.size()); }

And when I inspect the returned string the value for the ñ is -15 (which is 241-256) -- its out of range for a 8 bit signed char (which is 7 bits + the sign bit)...

Inside the debugger ñ is well displayed as ñ. All seems to be okay but when I load the string in a .NET application, the character ñ value is 65533 and it does not display as ñ!!

So it is an encoding problem!!
Juan Dent 226 Reputation points

2022-08-31T20:59:46.803+00:00

Looking at the character map application that is part of Windows 10 Pro, I can verify that ñ has a value of 241 in the unicode character set as well as the DOS : United States character set which is also 241
Laxmikant 121 Reputation points

2022-09-04T02:39:56.233+00:00

you can try

Convert.ToBase64String(Encoding.UTF8.GetBytes(dirtyString)))

https://geeksarray.com/blog/using-fluent-api-in-efcore-code-first

Answer 1

Muhammed Şeker 1

I advice to you use entityframework and be careful your database column variable types.

How to set text encoding on a data column from EntityFramework or ADO.NET?

1 answer

Activity