Is the "`n" option in FileAppend and FileRead confusing?

[iseahound]

It's just bizarre that I need to specify "`n" to produce a file with "`r`n" line endings.

A related issue is that a continuation section uses `n by default. So the example code:

Code: Select all

FileAppend("
(
Line 1
Line 2
)", "test.txt")

contains a `n between Line 1 and Line 2. Generally I expect the current CRLF / LF setting of the AHK file to be used instead, such that a normal windows script should emit CRLF/`r`n, unless the user has saved the file using LF/`n.

[just me]

I agree in relation to FileAppend. If there is an option to insert `r before `n it should be `r.

I was hoping that v2 would get rid of the (IMO) useless EOL conversions and consistently use `r`n as EOL default. For some reason the conversion rules have only been changed.

I don't mind at all because I don't use such options in v2.

[V2User]

I think use ClRf to replace 'n to represent 'r`n might be better. And use LineFeedByOS to follow the system's own setting.

[lexikos]

FileOpen was originally designed by jackieku (Kai-Chieh Ku) with separate options for translating `r`n and standalone `r to/from `n. I added character-based options, including `r and `n. For FileOpen, I think that `n being the option to translate both to and from `n is reasonable, and disagree that using `r for either FileOpen or FileAppend would be "less confusing". I'm guessing I chose to use the same character in FileAppend for consistency with FileOpen. It's surely better than * to disable translations in one function, *t to enable them in another, and `n in another.

FileAppend defaults to no end-of-line translations, consistent with FileRead and FileOpen. FileAppend and FileRead both have a separate Options parameter [...]. FileAppend, FileRead and FileOpen use "`n" to enable end-of-line translations.
Source: Changes from v1.1 | AutoHotkey v2

I was hoping that v2 would get rid of the (IMO) useless EOL conversions and consistently use `r`n as EOL default.

I can barely understand what you mean, so can only guess as to why such a change wasn't made. The best explanation would probably be that I didn't even consider it; I certainly wouldn't know what you're hoping for if you didn't tell me. There was a ten year window during which such a suggestion could have been made. (Maybe it was, but I don't remember it.)

Aside from EOL conversions (which are already disabled by default in most cases), I suppose that "consistently use `r`n as EOL default" would mean:

• Interpret `r`n as EOL in functions such as File.ReadLine, Loop Read, RegEx... These already interpret `r`n as EOL.
• Produce `r`n in multi-line strings such as A_Clipboard (filenames), WinGetText and RegRead (REG_MULTI_SZ). The first two already use `r`n.
• Default to Join`r`n in continuation sections.

// The default is linefeed because:
// 1) It's the best choice for hotstrings, for which the line continuation mechanism is well suited.
// 2) It's good for FileAppend.
// 3) Minor: Saves memory in large sections by being only one character instead of two.

`r`n has historically been sent as two individual Enter keystrokes (as per VkKeyScanEx for each character), although I changed that for Text mode (in v1). If `r`n was sent as a single Enter, `n vs `r`n wouldn't matter for hotstrings.

It's evidently not so good for FileAppend anymore, but even in v1, I can't see that Join`n is better for FileAppend given that it will end up as `r`n either way.

Prior to Unicode support, EOL translations in FileAppend and Loop Read were done automatically by the C runtime, so that may have influenced the default behaviour.

I would presume that EOL translations were desired because producing and parsing strings with `n is easier or more convenient than with `r`n.

[just me]

@lexikos
e.g. EOL On Off (End of LIne Translation).
I'm pretty sure there are other posts.

[iseahound]

I think consistency is the most important here.

1. Windows uses CRLF by default, so Windows users expect CRLF by default.

1a. This could mean AHK uses CRLF by default.

1b. OR this could mean that AHK uses the file encoding by default.

2. `n is probably better for parsing since it's a single character. If the functions in AHK could switch between `r`n or `n as the line ending default, that would be great. However, one thing that worries me is that if you use a simple herustic such as "detect `r`n", this might mess with the rare file that has mixed line endings?

3. Thanks for looking into this.

[RaptorX]

1. Windows uses CRLF by default, so Windows users expect CRLF by default.

I think we should ignore what Windows do in this regard

This is an old artifact that everyone and their grandmothers understood should be eliminated and Windows decided to stick to it for... reasons (probably compatibility reasons).

I think `n should be de default, and whenever me, as the developer, am forced by windows or other application to use `r`n then I would like to use an option when opening/saving the file.

I do agree with OP that `n meaning CRLF is extremely weird.
I think this makes more sense:

Code: Select all

FileAppend
(
"this is 
multiline
text that
will be appended"
), a_desktop "\test.txt", "`r"       ; add CR to each new line

[lexikos]

If someone comes up with a comprehensive list of affected functions and changes needed to implement a coherent solution that satisfies everyone... I will be shocked.

@just me, I don't see any suggestions in that topic resembling what you hoped for, not that it would matter if there were.

[RaptorX]

If someone comes up with a comprehensive list of affected functions and changes needed to implement a coherent solution that satisfies everyone... I will be shocked.

@just me, I don't see any suggestions in that topic resembling what you hoped for, not that it would matter if there were.

I dont think many functions are affected by this problem.

Only these three come to mind right now:

FileRead
FileAppend
FileOpen

Interestingly FileOpen behaves closer to what we are referring to, because the `n option Replaces `r`n with `n when reading a file. It does convert them back to `r`n on write, but the point here is that `n does the conversion to `n which is obvious and logical.

The other two on the other hand do something strange. When you specify the `n option you get `r`n which is somewhat unexpected.
What we are suggesting is that those two (I cant think of other functions that might benefit from this right now) take `r as the option to produce `r`n which seems a bit more logical.

For FileRead this has the side effect that now you dont have to worry about excluding `r in your loops unless you explicitly state you want the file to be opened with `r`n by specifying FileRead filePath, "utf-8", "`r" which now makes sense.

I am of the mindset that Autohotkey should default to `n when reading files unless told to do otherwise but that is not OP's suggestion.
At a glance it looks like a small change (change the `n option on those commands to be `r instead).

None of this is taking into consideration the technical issues that the change might pose (e.g. maybe breaks backward compatibility). So I take it more as a wish than anything else.

[lexikos]

The other two on the other hand do something strange. When you specify the `n option you get `r`n which is somewhat unexpected.
What we are suggesting is that those two (I cant think of other functions that might benefit from this right now) take `r as the option to produce `r`n which seems a bit more logical.

I think I've already explained this. The pairing of FileAppend and FileRead is roughly equivalent to FileOpen followed by File.Write and File.Read. The option for FileRead is the same as for FileAppend and for FileOpen. The same logic applies to both pairs. A conversion is done to or from `n.

You say that `r "seems a bit more logical", but you (and others) haven't provided any reasoning as far as I can tell. It seems to me that it is mostly a matter of intuition or opinion, unexplained and not backed by logic.

EOL handling is not just a choice of which character to use, but whether to perform conversions, which conversions to perform, and which line ending to use when a line is given (i.e. WriteLine, or the continuation section's default joiner). Each conversion has two sides, and there are possibly more than two alternatives on each side (if we bother with standalone `r).

Yet multiple participants in this topic have used wording like "`n should be the default" or "use `n by default". So when the focus is clearly on `n, why is anyone suggesting that using `r as the option character is more logical? This makes no sense to me.

(Why) would it not be unexpected to specify `r and get `r`n? Or to specify `r and get `n, or for specifying `r to affect how `n in the data is interpreted?

Do you have trouble remembering that the option is `n? Would it be easier to remember something more arbitrary, like t or a numeric bitflag? Would such options be more or less "logical" than `n?

How and why were those who answered "yes" to "is the `n option "confusing"?" actually confused by the option? Were they actually even confused, or just lacking an understanding of the motivation behind the choice of character, especially after the topic title has drawn their attention to it? Is such an understanding necessary to remember and use the option effectively?

Did this topic even have a practical purpose? There aren't any clear suggestions for change. The continuation section default not meshing with FileAppend's default behaviour is a clear issue, but that's a "related issue", tangential to the topic title.

[iseahound]

I've thought a bit about a proposed solution. The idea is to (1) maintain consistency when reading / writing to an outside source. And (2) to enforce source → destination flags. And finally (3) to make the options available to be as clear as possible.

1. Continuation Sections and FileOpen both "read from an outside source". Currently continuation sections use linefeeds only, while FileOpen maintains original line endings. I think this should be made consistent.
   • EITHER: Continuation sections should take into account the original source encoding of the script file. (#included files should have their encoding defaulted to the master script)
   • OR: FileOpen defaults to `n
2. FileRead uses source → destination, but FileAppend does not. For example, `n correctly replaces CRLF to LF when reading from an external file, but does the inverse when writing to the file.
   • As a consequence of this change, FileOpen should have an additional parameter. The 2nd parameter should dictate the line ending when reading from the script, and the 3rd parameter should dictate the line ending when writing to the script.
3. All arguments should be simple. `r means replace the line ending with a carriage return. `n means replace the line ending with a linefeed. `r`n means replace the line ending with the carriage return linefeed. There should be no "Other" meanings to any of these arguments, please use a different sequence of letters for anything else.
   • The user should never have to look up the manual to see what `r does in FileOpen.

[iseahound]

So a few examples,

Code: Select all

; This means open the file with LF line endings, but make sure to write CRLF. 
f := FileOpen("temp.txt", "rw`n", "`r`n")
f := FileOpen("temp.txt", "r`n", "w`r`n") ; Maybe read and write can go in any of the two fields?

Code: Select all

FileAppend("string", "temp.txt", "`r`n") ; Write CRLF line endings
FileAppend("string", "temp.txt", "`n") ; Write LF line endings
FileAppend("string", "temp.txt", "`r") ; Write CR line endings

Code: Select all

Text := FileRead("temp.txt", "`n") ; Convert all line endings to LF
Text := FileRead("temp.txt", "`r") ; Convert all line endings to CR
Text := FileRead("temp.txt", "`r`n") ; Convert all line endings to CRLF

This follows the current behavior of Join

Code: Select all

Join`r`n ; joins with CRLF
Join`r ; joins with CR
Join`n ; joins with LF

The only drawback is an extra parameter for FileOpen.

In the scenario where there are mixed line endings, the user should manually use StrReplace or RegExReplace to detect and handle individual cases for each line ending character: carriage return (CR), line feed (LF), and even form feed (FF), as well as the separators: File Separator (FS), Group Separator (GS), Record Separator (RS), and Unit Separator (US). See: https://en.wikipedia.org/wiki/C0_and_C1_control_codes

Unknown: should form feeds be converted to LF as well?

Q: How should FIleOpen know which line ending to use for WriteLine if a line ending isn't specified?
A: For simplicity, just search for the first instance of `r, `n, or `r`n. If none of these are present, the current encoding of the master script should be used instead?

[RaptorX]

A conversion is done to or from `n.

I think this is the part that we find a bit odd (at least me and everyone who selected "yes" on the poll). When I say FileAppend text, 'file.ahk', 'UTF-8' I am not converting from UTF-8, but rather to UTF-8.

So having `n convert from `n to `r`n feels odd because the expectation is that, if I use that option, my end result will have what I explicitly set for that option, so our expectation is that Fileappend text, 'file.ahk', '`n' would result in a `n delimited file. I can't think of a simpler way of explaining this.

How and why were those who answered "yes" to "is the `n option "confusing"?" actually confused by the option? Were they actually even confused, or just lacking an understanding of the motivation behind the choice of character, especially after the topic title has drawn their attention to it? Is such an understanding necessary to remember and use the option effectively?

I have noticed that most of the back and forth that you and I usually have, centers around the difference in thinking between a programmer and a non-programmer.

As a programmer you are looking at the conversation for efficiency and strict logic, while I usually refer to what people "perceive to be easier" for whatever reason. Most of the time, what im referring to, is not strictly logical, but by how this poll is going, the people who is saying "yes" to the poll are referring to the same thing Im referring to.

It might not be strictly logical in the way you are thinking about it, but i guess sometimes pure logic breaks some kind of intuition we have, and this is just one of those examples where this happens.

Im not saying you are wrong. Is just a fact that we are looking at it from a different perspective.

[RaptorX]

Code: Select all

; This means open the file with LF line endings, but make sure to write CRLF. 
f := FileOpen("temp.txt", "rw`n", "`r`n")
f := FileOpen("temp.txt", "r`n", "w`r`n") ; Maybe read and write can go in any of the two fields?

Code: Select all

FileAppend("string", "temp.txt", "`r`n") ; Write CRLF line endings
FileAppend("string", "temp.txt", "`n") ; Write LF line endings
FileAppend("string", "temp.txt", "`r") ; Write CR line endings

Code: Select all

Text := FileRead("temp.txt", "`n") ; Convert all line endings to LF
Text := FileRead("temp.txt", "`r") ; Convert all line endings to CR
Text := FileRead("temp.txt", "`r`n") ; Convert all line endings to CRLF

This follows the current behavior of Join

Code: Select all

Join`r`n ; joins with CRLF
Join`r ; joins with CR
Join`n ; joins with LF

Totally agree with those examples. This is in line with what im also thinking.
I was not really thinking about an extra parameter, but the end results according to those examples make sense to me.

I dont really know why you would like to have a `r only EOL but yeah, thats okay too.

I was envisioning something more like:

Code: Select all

FileAppend("string", "temp.txt", "`n") ; Write LF line endings
FileAppend("string", "temp.txt", "`r") ; Write CRLF line endings

Yes, I know, not 100% logical, but at least it made sense to me because as I said, I haven't found the need for a `r only EOL.
And FileAppend would obviously default to one or the other. My preference is `n but windows prefers `r`n so that would be up to you what the default for Autohotkey should be.

[V2User]

I suddenly agree with lexikos now, since I have realized that there are only linefeeds of two sides. One side is the OS side, which is always the default linefeed such as 'r'n in windows. While the other side is AHK side, which is probably 'n. Just like Python, it automatically converts linefeeds betweenPython side and OS side, with no need to know what's the default linefeed of the OS it is running on.

It will be more logical and reasonable, if the three func FileRead FileOpen FileAppend could use the same option, i.e. `n. There are probably three advantages for using `n:

(1) If the above three Funcs share the same value of an argument of them, code will become more consistent and more stable.

(2) They are platform compatible. If the three Funcs use `n only for the AHK side, codes of the three functions will never need change in the future, if they are migrated from one OS to another, such as Linux OS, just like what acts in Python.

(3) Preserve more flexibility to extend more support in the future, if `n is not the only thing in various AHK Strings. For example, using `r option for FileAppend() to automatically convert `r to `r`n, or using `r|`n option to convert `n,`r,`n`r to `r`n.

(4) If you want to change the default linefeed from `r`n to another for windows file, just omit the option and change the String manually, such as using Join `r or Join anything. There are no needs for FileRead FileOpen FileAppend to do these again. They are just out of what the three File Funcs should do. Since AHK's others can do these instead.

Keeping the above four in mind, I think keep `n is not confusing but more reasonable.

[RaptorX]

@V2User
So basically you mean to say that it makes total sense to you that if you say "hey, create a file and here's the `n option" AHK would say "sure, here's your file with `r`n EOL".

What Lexikos is arguing is that it is irrelevant what letter we use for that option, we should just understand that `n in this context means: convert the EOL to `r`n...

Is just as weird expecting that hipothetically Join`n would produce a string with `r`n because thats just what that option means, and after you learn it, there's no confusion anymore. (at least that's how I am understanding his comments).

You see? In reality Join`n produces a string with `n but FileAppend with the `n option gives you a file with `r`n... And that's logical somehow. So, if you truly agree with Lexikos, explain how that is NOT confusing.

I understand what he is thinking, he has explained himself several times and what he is saying has its merits, I just don't agree with it. Because it means that `n has different meanings depending on the context and I think that shouldn't be like that.

Case in point, from the documentation:

FileRead

Options
Type: String

`n (a linefeed character): Replaces any/all occurrences of carriage return & linefeed (`r`n) with linefeed (`n). (yep, totally expected and fair)

FileOpen

End of line (EOL) options
Flag Dec Hex Description
`n 4 0x4 Replace `r`n with `n when reading and `n with `r`n when writing. (meh, i dont like it but i understand the logic here)

FileAppend

Options
Type: String

`n (a linefeed character): Inserts a carriage return (`r) before each linefeed (`n) if one is not already present. (ok... lol what?!)

There is no consistency on what `n means between commands, and we are arguing, there should be. We shouldn't have to remember what `n means on each of the commands. If you change FileAppend's option to be EOL to mean "EOL conversion" it would make even more sense to me than `n having that meaning.

The option for FileRead is the same as for FileAppend and for FileOpen. The same logic applies to both pairs. A conversion is done to or from `n.

I understand that you mean that the `n option does a conversion in the three commands, but I think the results of that conversion varies between them, and that, is what is confusing to some of us.

And in the end I think only FileAppend is the weird one in this list.

[lexikos]

Any solution must account for backward-compatibility. For instance, continuation sections cannot change behaviour solely based on which line ending is used in the source file, because scripts already primarily use `r`n but have continuation sections which produce `n, and the behaviour is explicitly documented. The meaning of `n in FileOpen's second parameter cannot be changed; it already enables conversion both ways.

I think this is the part that we find a bit odd (at least me and everyone who selected "yes" on the poll). When I say FileAppend text, 'file.ahk', 'UTF-8' I am not converting from UTF-8, but rather to UTF-8.

You previously suggested to use `r as the option for producing `r`n, and yet "UTF-8 `r" would not convert to UTF-8 with `r as a line ending.

FileAppend would obviously default to one or the other. My preference is `n

The default behaviour is to write whatever is given. If you want `n, you use that in the data being passed to the function. As a default behaviour, this has the least potential for surprise, and is the most efficient way to permit every kind of line ending. Defaulting to `n implies that the function will interpret and convert line endings by default. This may be unwanted, even if contrary behaviour hadn't already been established and documented.

There is no consistency on what `n means between commands

The inconsistency you have demonstrated is merely in the wording of the documentation, not the meaning of the option.

FileAppend, FileRead and FileOpen take data between memory and file. In what terms could you explain an option as having the same meaning in all of these contexts? You must either eliminate the differences between the contexts, or account for them. The difference between them is that FileAppend writes to file, FileRead reads from file, and FileOpen could do either. The commonality is that when the `n option is used, `r`n is in the file and `n in memory. In other words, `n means to convert between `n (in memory) and `r`n (in the file).

In reality Join`n produces a string with `n but FileAppend with the `n option gives you a file with `r`n... And that's logical somehow.

Each option is logical within the appropriate context. Nothing can be taken out of context and retain its original meaning perfectly, by definition:

context: the circumstances that form the setting for an event, statement, or idea, and in terms of which it can be fully understood.

What is not logical, is using a comparison between continuation sections (a language construct) and file I/O functions to argue that options in these disparate constructs aren't logical. You aren't just removing context, but changing the context to one which makes less sense.

I have noticed that most of the back and forth that you and I usually have, centers around the difference in thinking between a programmer and a non-programmer.

There are many more than two different ways of thinking. There are also non-programmers who properly exercise logic, and programmers who don't.

Most of the time, what im referring to, is not strictly logical

To me, "not strictly logical" is just irrational. You could arrive at a conclusion by intuitive leap, yet still rationalise and explain it with logic. If you don't understand why you arrived at a conclusion, how can you expect to communicate understanding to someone else?

Logic is an important part of communicating effectively, especially in a debate.

I was asking about the purpose of this topic, questioning whether there is actually a practical reason to change the `n option. I fail to see what that has to do with thinking like a "programmer".

What Lexikos is arguing is that it is irrelevant what letter we use for that option

I don't think I did.

For example, `n correctly replaces CRLF to LF when reading from an external file, but [and correctly] does the inverse when writing to the file.

Writing is the inverse of reading.

For simplicity, just search for the first instance of `r, `n, or `r`n. If none of these are present, the current encoding of the master script should be used instead?

Simplicity is to have a single line ending value which is either a known default or whatever the caller explicitly specified. Searching the file (if the file is even being opened with read access) adds complexity. Providing a default based on what line ending the script file uses adds more complexity, and potential for error. I see no reason to assume any there is any relation between the encoding of the script file and the encoding of the files it is processing.

I think the results of that conversion varies between them

In what way?

[iseahound]

It's not possible to have a clean solution if backwards compatibility is taken into account. The current meaning of `n has three legs: CRLF → LF → CRLF which becomes confusing when only using FileRead CRLF → LF or FileAppend LF → CRLF. It's just better if `n means what it means.

[RaptorX]

You previously suggested to use `r as the option for producing `r`n, and yet "UTF-8 `r" would not convert to UTF-8 with `r as a line ending.

I'm suggesting that `r should insert `r in front of each new line. As opposed to `n inserting `r in front of each new line.

What is not logical, is using a comparison between continuation sections (a language construct) and file I/O functions to argue that options in these disparate constructs aren't logical. You aren't just removing context, but changing the context to one which makes less sense.

That is exactly my point. When writing code, context changes very often, in one line I'm creating a continuation section, in the next one I'm file appending. The fact that it makes "less sense" when we change the context is basically the point of the whole thread and why we want to change it so that it makes a "bit more sense".

There are many more than two different ways of thinking. There are also non-programmers who properly exercise logic, and programmers who don't.

Completely agree.

To me, "not strictly logical" is just irrational. You could arrive at a conclusion by intuitive leap, yet still rationalise and explain it with logic. If you don't understand why you arrived at a conclusion, how can you expect to communicate understanding to someone else?

Well, humans arent logical. We are quite irrational most of the time. I am trying to convey how that irrationality shows up in different parts of the language. It is really difficult to explain. You are biased toward strict logic. But when we are coding we dont really think strictly logically all the time.

Logic is an important part of communicating effectively, especially in a debate.

Totally agree with you there.

I was asking about the purpose of this topic, questioning whether there is actually a practical reason to change the `n option. I fail to see what that has to do with thinking like a "programmer".

I think you are missing my point: it has a lot to do with NOT thinking like a programmer.

As a programmer I totally agree with your assessment of the situation, and how you'd have to account for backward compatibility.
I also agree that with the correct context everything "makes sense". I am just arguing that what OP brought up is a valid confusion that I think you are just dismissing without taking into account the way how non-programmers think about coding.

I don't think I did.

Not directly but that is what you are suggesting when you say:

(Why) would it not be unexpected to specify `r and get `r`n? Or to specify `r and get `n, or for specifying `r to affect how `n in the data is interpreted?

Do you have trouble remembering that the option is `n? Would it be easier to remember something more arbitrary, like t or a numeric bitflag? Would such options be more or less "logical" than `n?

In the end it sounds like you are saying: Once you understand what `n does in that context, then it shouldn't confuse you.
Sadly that is not how the human brain works. We are in fact, very irrational

I'm in favor of structuring the language in a way that we can predict what will happen without having to read the manual at each turn.

e.g.: WinMove vs GuiCtrl.Move() I wasted a few hours just because those two behave differently under certain specific circumstances... those little details makes us waste too much time. You might have a perfectly valid and logic argument as to why they should behave that way, but in practice they become time wasters because we have to go and check the manual every time.

Writing is the inverse of reading.

I hope you are not suggesting that contents := FileRead('path/to/file', 'UTF-8') would read an ANSI file and have it as UTF-8 in memory but that
FileAppend contents, 'path/to/file', 'UTF-8' would save it back to an ANSI file, because, you know, writing is the inverse of reading.

And this is basically the core of the problem with the `n command in FileAppend, It doesn't behave like the other options in that same command. UTF-8 Saves the file as UTF8, RAW writes the bytes as-is, but `n does the inverse of FileRead's `n option.

I think the results of that conversion varies between them

In what way?

FileRead
`n results in `n

FileAppend
`n results in `r`n

FileOpen
`n results in both `n and `r`n depending on whether you are reading or writing

Thus the results vary.

[Marium0505]

This sounds more like a v3 discussion than a v2 discussion to me as in my opinion, backwards compatibility should be one of the top priorities.. Otherwise it will only get even more confusing if things behave differently from one v2 version to another.

How about adding a new function though? Then current functions will behave as they do today (thus being backwards compatible), and the discussed changes instead added to the new function. The best of two worlds?
And then improve the documentation to reduce confusion on how the current function(s) actually behave, in most common situations.