Powershell to remove special characters in text file

Question

Powershell to remove special characters in text file

RJ 326

Hi there,

On executing the below script with the attached input file, looking for help to remove the special characters. How to remove them but single quotes need to be the same. You could see some special characters in output line 9,10,11,12

output

["Continental District â€“ Denver", "Org Unit"],
["The Team Leadâ€™**s Roadmap", "Org Unit"], --<<The single quotes in the word Lead's need to be retained>>
["Data ServicesÂ** - ABC", "Stem Face"],
["App tttmÂ - CST", "Stem Face"],

script used

$file = Get-Content -Path "C:\temp\pinput.txt" -Raw
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}'
$file |Out-File -FilePath "C:\temp\oput.txt"

155372-pinput.txt
Thanks.

Accepted answer

0 additional answers

Your answer

Answer 1

DaveK 1,871

get-content C:\temp\pinput.txt -Encoding UTF8 | Set-Content c:\temp\poutput.txt

I believe the output from this command retains the single quote but removes all other special characters.

RJ 326 Reputation points

2021-12-06T20:15:22.97+00:00

Thanks. The above works as expected. Any suggestion on how to integrate this into the existing above code?

Is there a way to

$file = Get-Content -Path "C:\temp\pinput.txt" -Raw # when i use Encoding here, the logic does not work. You might notice [abcd] is broken into 2 lines in input file and the logic fixes it by bringing [abcd] in one line]
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}'
$file # can i read the content of this by using -Encoding UTF8 and write to output? i tried something like below but if fails. i'm sorry im new to ps. appreciate your help.
$file = $file -Encoding UTF8 | Set-Content c:\temp\poutput.txt

Thanks
Rich Matheisen 47,901 Reputation points

2021-12-06T20:32:48.833+00:00

Are you using the "-Raw" and "-Encoding UTF8" together? Or are you replacing "-Raw" with "-Encoding UTF8"?
RJ 326 Reputation points

2021-12-06T21:52:50.697+00:00

I replaced -Raw with -Encoding UTF8. When i do that, the existing logic of abcd (split as 2 lines in input and logic fixes as single line) does not work for some reason when i use -encoding utf8.

May be im missing something?
Rich Matheisen 47,901 Reputation points

2021-12-06T22:24:00.66+00:00

Don't replace "-Raw", just add "-Encoding UTF8" to the Get-Content.

By replacing the "-Raw" switch you're getting an array of strings instead of one string the holds the entire contents of the file (including the end-of-line character(s)).
DaveK 1,871 Reputation points

2021-12-07T19:05:16.67+00:00
I'll admit to have a very limited understanding of regex but even with just your code and no alterations, I can't see the regex effecting the output in anyway. You mention the abcd line which is spilt which I believe in the source file is line 7 & 8, when running I'm not seeing any formatting change between the value of $file after reading the source file and once the regex is applied (see image)

Is it possible you could maybe provide an example of what you would expect the full poutput.txt to look like after running your script?

If you are able to get the required output from your regex without using the -Encoding UTF8 option, can you not just run the output from the fixed formatting (bring abcd back together etc) and do another open/save using the UTF8 encoding?

$file = Get-Content -Path "C:\temp\pinput.txt" -Raw $file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}' $file | Out-File -FilePath "C:\temp\poutput.txt" $file = get-content "C:\temp\poutput.txt" -Encoding UTF8 $file | Set-Content "C:\temp\poutput.txt" -force
Rich Matheisen 47,901 Reputation points

2021-12-07T19:59:36.217+00:00

His intention is to drop the intervening newline (CrLf) between the two patterns. That would join the two patterns on a single line.

[ab<newline>
cd]

would become

[abcd]

He's hasn't mentioned that this is a continuation of an earlier post he made about how to accomplish that task. The regex needs work. His original post only asked to combine the lines when a line began with "[" and didn't end with "]" and was followed by a line that did not begin with "[" but did end with "]". He later added additional conditions and said he was satisfied with his own solution.
RJ 326 Reputation points

2021-12-09T14:30:23.2+00:00

Thank you Rich. My bad. I was replacing -Raw. Per your above recommendation by adding encoding it works s expected. I wonder why im not able to mark yours as the answer. Thank you for your help.

I'm trying to modify and learn how to trim the leading spaces before the second line (if any -add space before the 2nd line cd)

Doesnot work
$file = Get-Content -Path "C:\temp\pinput.txt" -Raw
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}.trim()'
$file |Out-File -FilePath "C:\temp\oput.txt"
Rich Matheisen 47,901 Reputation points

2021-12-09T15:15:22.83+00:00

First, I think you should explain what your regex is doing, especially the first argument of the "-replace" operator.
Second, the 2nd argument of the "-ireplace" is surrounded by single quotes. Is that really what you want????

Third, a question can have only a single answer in this system. You can, however, "Vote" for a comment.

Lastly, you should really post another question regarding the regex. The regex has nothing to do with the topic of your original post.
RJ 326 Reputation points

2021-12-09T16:52:31.823+00:00

Thanks Rich. I will post it as another thread. I will vote yours as well.

I came across regular expression "captured groups" or "groups capture". Im sure you might be aware of that. My understanding is captured groups use single quotes per syntax. I tried to build and play around it.

first group checks for [" and the 2nd checks if there is no "] on the same line, then it concatenates next line. im new so still reiterating it.

finds for [" and "] - works fine.
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}'

One thing i couldnt understanding is how to remove leading space of 2nd line before contcatenate. below doesnt work.
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}.trim()'

Thank you very much Rich.
RJ 326 Reputation points

2021-12-09T17:02:16.83+00:00

My bad Dave. Not sure if it was copy paste issue. Now encoding issue is resolved per yours and Richs' suggestion. So marked the thread as answered. Thanks for your help.

The below is the correct code.. if you give it a shot it will show the right way.

$file = Get-Content -Path "C:\temp\pinput.txt" -Raw
$file = $file -ireplace '(?<match1>[\"[^]])\r\n(?<match2>[^]]\"])','${match1}${match2}'
$file

input: (pattern is to find only [" and the same line does not contain "] anywhere in that line then contact the next line. Other lines to be as is. Now if 2nd line has leading spaces, i'm not able to remove it.
["ab
cd"]

expected output
["abcd"]

Thanks a lot for your help.

Share via

Powershell to remove special characters in text file

0 additional answers

Your answer