Unicode and ASCII may not be encoding correctly

Question

Hi, I am using visual studio 2022 to host a .NET 8 MVC web application project. I am trying to read a text file and write it to another text file.

   This is the code I am using:

string wdata = System.IO.File.ReadAllText(file);          

 // This is the statement that determines the encode
 // result in the output text file

 System.IO.``File.WriteAllText(wfile, wdata, System.Text.Encoding.Unicode);

  The problem is that the text file I am reading from has ”hyphens”. But when I read it, and write it to the other text file, the ”hyphens” turn into:

This is the “hyphen” in the original text I am trying to read.

This is when written using the encoding ASCII

This happens when written with encoding UTF8

Thanks for any advice or suggestions

Answer 1

Hi,

Since this thread has already provided a lot of insights, I would like to give a better summarization of this problem and the approach to solve it.

---

This problem occurs because the text file being read contains characters (such as en-dash, em-dash, curly quotes, etc.) that are not part of ASCII. The file itself was originally saved in a single-byte code page (often Windows-1252 / “ANSI”). When it’s read without explicitly specifying the correct encoding and then written out with a different one (e.g., Unicode/UTF-16), the characters are misinterpreted and show up incorrectly.

In short:

• Mismatched encodings cause the corruption: reading with one code page, writing with another.
• ASCII cannot represent extended symbols like en-dash or curly quotes.
• .NET’s File.ReadAllText defaults to UTF-8 in .NET Core/.NET 5+ and to the system ANSI code page in older .NET Framework apps, so behavior may differ across environments.

---

Ways to Resolve

1. Read/Write with the Correct Legacy Encoding If your file was created in Windows-1252 (ANSI), explicitly specify that when reading/writing:

      Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
      string input = File.ReadAllText("input.txt", Encoding.GetEncoding(1252));
      File.WriteAllText("output.txt", input, Encoding.GetEncoding(1252));
   

2. Migrate to Unicode (Recommended) After detecting the correct source encoding, convert to UTF-8 or UTF-16 to ensure all characters are preserved consistently:

      string input = File.ReadAllText("input.txt", Encoding.GetEncoding(1252));
      File.WriteAllText("output.txt", input, Encoding.UTF8);
   

3. Avoid Encoding.Default Since it varies by OS and runtime, relying on the default encoding can lead to inconsistent results across machines.

---

In conclusion: either use the same encoding for both reading and writing (e.g., Windows-1252) or convert your files to a Unicode encoding like UTF-8 for future use.

Hope this helps.

Answer 2

I found this fix:

// Read Text test - Read the text from a txt file and add create a new txt file

  // with the original txt file data

  public void ReadAndWriteTextTest()

  {

      // Get the root path

      var webRoot = _env.WebRootPath;

      // Text with this txt file which is a copy of the MS Word document

      // what_we_do_and_how.docm

      var inputfile = System.IO.Path.Combine(webRoot, "What_we_do_and_how ENCODED.txt");

      // The txt file name to be created.

      var outputfile = System.IO.Path.Combine(webRoot, "A6CWrite.txt");

 

      // RC@202503100000 - Get the 1252 specific code page to use with Encoding.RegisterProvider

      Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

            

      // Read the input txt file data

      string inputdata = System.IO.File.ReadAllText(inputfile, System.Text.Encoding.GetEncoding(1252));

            

     // Create and write to the ouput txt file

      System.IO.File.WriteAllText(outputfile, inputdata, System.Text.Encoding.GetEncoding(1252));

 

 }

the outputfile now shows the correct encoding :

Answer 3

I don't think that your "hyphen" in the text file belongs to ASCII.

Please open the text file using binary editor available in Visual Studio 2022.

If your "hyphen" really belongs to ASCII it will be shown as "2D":

Answer 4

Hi @iqworks Information Quality Works

Try reading the file using Encoding.UTF8 explicitly to ensure that special characters are preserved：

string wdata = System.IO.File.ReadAllText(file, System.Text.Encoding.UTF8); System.IO.File.WriteAllText(wfile, wdata, System.Text.Encoding.UTF8);

I would recommend loading the file into a text editor like Notepad++ that tells you what the encoding of the file is (in the status bar). If it's not encoded in UTF-8 to start with, reading and writing as UTF-8 won't work.

---

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Best regards,

Rena