How to remove hidden end of line (EOL) in text

[songdg]

I use the Loop (read file contents) method to read a txt file's content, it only contain a single line, and should be just one loop, but somehow the text be seperated into 3 sections. It seems some hidden end of line (EOL) in the text file.
Here's the text filehttps://www.udrop.com/shared/8rjk3lwi_b ... uz3qb-74np

[songdg]

Since I can't upload an attachment, I post the conten here,

该产品中“化瘀”一词，涉嫌虚假广告。
商品网址：
https://detail.tmall.com/item.htm?spm=a230r.1.14.241.47d946c5ni3KV6&id=625007972071&ns=1&abbucket=8&skuId=4433074353231

[AHKStudent]

I can't test it but if you try

Code: Select all

FileRead, myFile, filename.txt
Msgbox, % myFile  ; what happens? 
Loop, parse, myFile, `n
     msgbox, % a_loopfield ; what happens?

[songdg]

I can't test it but if you try

Code: Select all

FileRead, myFile, filename.txt
Msgbox, % myFile  ; what happens? 
Loop, parse, myFile, `n
     msgbox, % a_loopfield ; what happens?

Just copy the text into a txt file's first line and save.

[songdg]

I can't test it but if you try

Code: Select all

FileRead, myFile, filename.txt
Msgbox, % myFile  ; what happens? 
Loop, parse, myFile, `n
     msgbox, % a_loopfield ; what happens?

Just copy the text into a txt file's first line and save.

The text has been changed throught copy and paste, It seems some hidden end of line (EOL) in my oringinal file.

[songdg]

Update the attachment

[sooyke_]

Code: Select all

FileRead, myFile,text.txt
myfile:=regexreplace(myfile,"\R")

tooltip, % myfile ; instead of msgbox to see a long line

FileAppend,%myfile%,remove_eol.txt
     
msgbox saved in remove_eol.txt

to see non printable characters thanks to Gregster :
https://www.soscisurvey.de/tools/view-chars.php

[garry]

@sooyke_ works fine , maybe possible also to use HEX-Editor

ascii 13 CR 0D
ascii 10 LF 0A

Example file 'text.txt' contains HEX 0A after each line and at end of last-line 0D 0A

Line1 0A
Line2 0A
Lastline 0D 0A

A question :
how remove only 0A , but keep 0D 0A ?

[sooyke_]

Code: Select all

FileRead, myFile,text.txt
myfile:=regexreplace(myfile,"s)\n(?=.*\n)")

tooltip, % myfile ; instead of msgbox to see a long line

MsgBox % myfile "`n" AscToHex(myfile) 


AscToHex(str) {
  Return str="" ? "":Chr((Asc(str)>>4)+48) Chr((x:=Asc(str)&15)+(x>9 ? 55:48)) AscToHex(SubStr(str,2))
}

[garry]

@sooyke_ your example works, made a test , I think after "How are you?" and " -ABCDE" should be CR ( 0D ) + LF (0A) , but keeps only CR ( 0D )
thank you for Asc2Hex
also interesting collection from user @jNizM , I used his Hex2Str ( Hex2Asc ) function ( I made it complicated )
https://www.autohotkey.com/boards/viewtopic.php?f=6&t=3514

Code: Select all

;- https://www.autohotkey.com/boards/viewtopic.php?f=76&t=92363
Fileencoding,UTF-8
F1=%a_scriptdir%\Test1.txt
F2=%a_scriptdir%\Test2.txt
ifexist,%f1%
  filedelete,%f1%
ifexist,%f2%
  filedelete,%f2%
transform,CR ,chr,13   ;- 0D  CarriageReturn
transform,LF ,chr,10   ;- 0A  LineFeed
transform,EOF,chr,26   ;- 1A  EndOfFile
cc:="Hello " . a_username . LF . " . How are you?" . CR . LF . "Newline-2" . LF . " -ABCDE" . CR . LF . "END" . CR . LF . EOF
fileappend,%cc%,%F1%,UTF-8
sleep,1000
FileRead,aa,%F1%
aa:=regexreplace(aa,"s)\n(?=.*\n)")     ;- script from user sooyke_  
bb:=AscToHex(aa)
;bb:="48656C6C6F204741525259202E20486F772061726520796F753F0D4E65776C696E652D32202D41424344450D454E440D0A1A"
i=0
loop,parse,bb
 {
 i++
 y:=a_loopfield
 M:=mod(i,2)
 if m=0
   e .= "0x" . z . y . " "
 z:=y
 }
ee:=HexToStr(e)
fileappend,%aa%`r`n`r`n-------- HEX : ----------`r`n%bb%`r`n`r`n-------HEX2STR : -----------`r`n%ee%,%F2%,UTF-8
run,%F2%
Exitapp
;-------------------------
AscToHex(str) {
  Return str="" ? "":Chr((Asc(str)>>4)+48) Chr((x:=Asc(str)&15)+(x>9 ? 55:48)) AscToHex(SubStr(str,2))
}

;- userx jNizM collection
;- https://autohotkey.com/boards/viewtopic.php?f=6&t=3514
;- Just a collection of small ahk functions / Feel free to add more
;-------------------------
hexToStr(hex)
{
    static u := A_IsUnicode ? "_wcstoui64" : "_strtoui64"
    loop, parse, hex, " "
    {
        char .= Chr(DllCall("msvcrt.dll\" u, "Str", A_LoopField, "Uint", 0, "UInt", 16, "CDECL Int64"))
    }
    return char
}
;MsgBox % "Hex:`t0x41 0x75 0x74 0x48 0x6f 0x74 0x6b 0x65 0x79`nHex:`t" hexToStr("0x41 0x75 0x74 0x48 0x6f 0x74 0x6b 0x65 0x79")
;===============================================================================

[sooyke_]

Hi Garry

Sorry , i misunderstood your question , i thought leaving only the last CRLf .............

Thanks for the example creating a test file with those CR and LF and EOF characters ! Added to my collection !

The AscToHex is an old one , i used it often .It was created by Laszlo ( wonder if he is still looking at the forum )
A takedown of this mindboggling code is here : https://www.autohotkey.com/boards/viewtopic.php?t=5576

Hope this regex works the way you descripted by leaving 0D0A unchanged.

Code: Select all

;- https://www.autohotkey.com/boards/viewtopic.php?f=76&t=92363
Fileencoding,UTF-8
F1=%a_scriptdir%\Test1.txt
F2=%a_scriptdir%\Test2.txt
ifexist,%f1%
  filedelete,%f1%
ifexist,%f2%
  filedelete,%f2%
transform,CR ,chr,13   ;- 0D  CarriageReturn
transform,LF ,chr,10   ;- 0A  LineFeed
transform,EOF,chr,26   ;- 1A  EndOfFile
cc:="Hello " . a_username . LF . " . How are you?" . CR . LF . "Newline-2" . LF . " -ABCDE" . CR . LF . "END" . CR . LF . EOF
fileappend,%cc%,%F1%,UTF-8
sleep,1000
FileRead,aa,%F1%
;aa:=regexreplace(aa,"s)\n(?=.*\n)")     ;- script from user sooyke_  
aa:=regexreplace(aa,"(?<!\r)\n")

bb:=AscToHex(aa)
;bb:="48656C6C6F204741525259202E20486F772061726520796F753F0D4E65776C696E652D32202D41424344450D454E440D0A1A"
i=0
loop,parse,bb
 {
 i++
 y:=a_loopfield
 M:=mod(i,2)
 if m=0
   e .= "0x" . z . y . " "
 z:=y
 }
ee:=HexToStr(e)
fileappend,%aa%`r`n`r`n-------- HEX : ----------`r`n%bb%`r`n`r`n-------HEX2STR : -----------`r`n%ee%,%F2%,UTF-8
run,%F2%
Exitapp
;-------------------------
AscToHex(str) {
  Return str="" ? "":Chr((Asc(str)>>4)+48) Chr((x:=Asc(str)&15)+(x>9 ? 55:48)) AscToHex(SubStr(str,2))
}

;- userx jNizM collection
;- https://autohotkey.com/boards/viewtopic.php?f=6&t=3514
;- Just a collection of small ahk functions / Feel free to add more
;-------------------------
hexToStr(hex)
{
    static u := A_IsUnicode ? "_wcstoui64" : "_strtoui64"
    loop, parse, hex, " "
    {
        char .= Chr(DllCall("msvcrt.dll\" u, "Str", A_LoopField, "Uint", 0, "UInt", 16, "CDECL Int64"))
    }
    return char
}
;MsgBox % "Hex:`t0x41 0x75 0x74 0x48 0x6f 0x74 0x6b 0x65 0x79`nHex:`t" hexToStr("0x41 0x75 0x74 0x48 0x6f 0x74 0x6b 0x65 0x79")
;===============================================================================

[garry]

Hi sooyke_ , thank you for solution

instead of
transform,CR ,chr,13 > can also use escapechar `r

https://www.autohotkey.com/docs/misc/EscapeChar.htm

Code: Select all

    HEX DEC
`n  0A   10   newline (linefeed/LF) 
`r  0D   13   carriage return (CR) 
`b  08   08   backspace 
`t  09   09   tab (the more typical horizontal variety) 
`v  0B   11   vertical tab -- corresponds to Ascii value 11. It can also be manifest in some applications by typing Control+K. 
`a  07   07   alert (bell) -- corresponds to Ascii value  7. It can also be manifest in some applications by typing Control+G. 
`f  0C   12   formfeed --     corresponds to Ascii value 12. It can also be manifest in some applications by typing Control+L. 

here I used  > transform,EOF,chr,26   :
    1A   26   End Of File (EOF)     Control+Z     

------------------
https://autohotkey.com/board/topic/29293-closed-collection-of-beautiful-one-liner-codes/page-2
-- [CLOSED] Collection of beautiful one-liner codes - Page 2 - Offtopic - AutoHotkey Community
------------------
https://www.autohotkey.com/boards/viewtopic.php?t=5576
-- asctohex function how does it work ? - AutoHotkey Community
------------------

[songdg]

Thanks all of you guys for providing the solutions, I simply use StrReplace() with "" replacing "`n" solve the problem.

[garry]

@songdg yes , this is correct
you had to remove `n to see one line , at last line you had `r`n , then > `r ( I mean `r is left )
my question was how to remove only `n but keep `r`n

Code: Select all

f1=%a_scriptdir%\text.txt
f2=%a_scriptdir%\textnew.txt
fileread,aa,%f1%
stringreplace,ab,aa,`n,,all
fileappend,%ab%,%f2%,utf-8
;- at end of text  is `r 
return

[songdg]

@songdg yes , this is correct
you had to remove `n to see one line , at last line you had `r`n , then > `r ( I mean `r is left )
my question was how to remove only `n but keep `r`n

Code: Select all

f1=%a_scriptdir%\text.txt
f2=%a_scriptdir%\textnew.txt
fileread,aa,%f1%
stringreplace,ab,aa,`n,,all
fileappend,%ab%,%f2%,utf-8
;- at end of text  is `r 
return

Sorry, my skill about AHK is just entry level.

[garry]

I'm not good in programming ... user sooyke_ had the solution for my special question ( remove `n , but keep `r `n )
for your solution was enough to remove `n
can use a HEX-Editor to see the special characters like `r `n `t ....