Why you shouldn't use FileAppend to continuously write to the disk

Put simple Tips and Tricks that are not entire Tutorials in this forum
DRocks
Posts: 441
Joined: 08 May 2018, 10:20

Re: Why you shouldn't use FileAppend

10 Feb 2019, 09:31

Ok cool. Thanks for taking time for such a clear answer.
I understand now. My current needs are always with small file sizes so I admit never tinking about the size problem with memory.

That loop example is quite interesting.
So for what I understand, small filesize would be totally fine for using just one variable and 1 file write. But as soon as you deal with bigger content its wise to care about memory loads.
Meroveus
Posts: 11
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

11 Feb 2019, 00:48

As I mentioned on Discord (username there is maugr1s) I tried to use the technique where you open the file at the beginning, write several times throughout the script, then close it at the end.

When I do this, the file shows in the directory as 0 length and cannot be read by an external program until it is closed.
I opened the file with the "a" flag

I tested a crash by using pskill to kill all instances of autohotkey.exe - which bombs the running script out of existence.
Under such circumstances, the buffer is not flushed, and the contents of the log are lost.

However, in the examples for the FileOpen() function is this piece of code:

Code: Select all

; Open a console window for this demonstration:
DllCall("AllocConsole")
; Open the application's stdin/stdout streams in newline-translated mode.
stdin  := FileOpen("*", "r `n")  ; Requires [v1.1.17+]
stdout := FileOpen("*", "w `n")
; For older versions:
;   stdin  := FileOpen(DllCall("GetStdHandle", "int", -10, "ptr"), "h `n")
;   stdout := FileOpen(DllCall("GetStdHandle", "int", -11, "ptr"), "h `n")
stdout.Write("Enter your query.`n\> ")
stdout.Read(0) ; Flush the write buffer.
query := RTrim(stdin.ReadLine(), "`n")
stdout.WriteLine("Your query was '" query "'. Have a nice day.")
stdout.Read(0) ; Flush the write buffer.
Sleep 5000
When I include the line:

Code: Select all

logFile.Read(0)
after the writes, the buffer is flushed, and the crashes don't lose data.

I would rather lose performance than data.

Thanks for your help!
User avatar
nnnik
Posts: 4037
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend

11 Feb 2019, 01:06

Yeah that is a problem - I wanted to make a few tests myself but got side-tracked at the end. From what I understood from the docs a, w, rw and r includes share flags so the locking behavior seems rather strange to me. Which system are you on?
Edit: Now I tested it myself and the buffer is indeed not flushed when I crash the script - I guess this is to prevent the program from writing bad data to the file - I need to change some stuff in the topic.
When Opening with "w" mode I was able to read the data from another program while it was still opened in AutoHotkey - same goes for the "a" mode.
Here is the script I used:

Code: Select all

file := FileOpen("crash.log", "w")
file2 := FileOpen("crash2.log", "w")

Loop {
	file.writeLine(A_Index)
	file2.writeLine(A_Index)
	file2.read(0)
	if (randomChance(1/100000)) {
		Msgbox about to crash
		causeCrash()
	}
}

randomChance(chance := 0.5) {
	Random,  randThrow,  0.0, 1.0
	return (randThrow<chance)
}

causeCrash() {
	causeCrash()
}
Recommends AHK Studio
Meroveus
Posts: 11
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

11 Feb 2019, 12:55

I'm on Windows 10 1803 64 bit.
I modified this somewhat by using:

Code: Select all

logFile := FileOpen(logPath,"a-d",CP1252)
to open the file.
I need it to append, I could use rw-d and position the pointer to the end, but it makes no difference
The -d prevents the file being deleted while logging. Seems prudent.
I use the alternate code page for convenience -- elsewhere the script uses UTF-8

Using the line

Code: Select all

logFile.Read(0)
To flush the cache seems reliable enough. In several tests the data survived intact.
Perhaps this could be documented in the section for the Read Method.

What method do you use to deliberately crash the script?
Meroveus
Posts: 11
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

11 Feb 2019, 17:43

Meroveus wrote:
11 Feb 2019, 12:55
What method do you use to deliberately crash the script?
Ha! Slick -- I just figured it out. I thought CauseCrash() might be in a library or something.

So the take-aways are:
  • Use file objects rather than file commands
  • Open once, write many, close once (on exit)
  • use Read(0) method following writes to flush buffer to prevent data loss.
  • lock file for deletion to prevent data lost due to inadvertent deletion while executing
All the best...
--M
User avatar
nnnik
Posts: 4037
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend to continuously write to the disk

19 Feb 2019, 05:40

I have included the newly gathered information.
Thanks for all the intereting and nice replies :)
Recommends AHK Studio
0x00
Posts: 69
Joined: 22 Jan 2019, 13:12

Re: Why you shouldn't use FileAppend to continuously write to the disk

23 Feb 2019, 00:22

Thanks for the insight, I've resolved to not use FileAppend as much...

FileAppend() & FilePrepend() incorporating your insights...

Code: Select all


Loop 10{	;file is only prepended to once & appended to the remainder of the loop,as FileAppend keeps the file open.
	FilePrepend("D:\Ut.txt", A_Now)
	FileAppend("D:\Ut.txt", "`n" A_Now " - Appended",true)	;keeps file open & also prevents other writes to file...
}
FileAppend("D:\Ut.txt")	;release file


Loop 10{	;file is both prepended & appended to
	FilePrepend("D:\Ut.txt", A_Now "loop2")
	FileAppend("D:\Ut.txt", "`n" A_Now " - Appended")
}

run, D:\Ut.txt
exitapp

FilePrepend(filePath, data){
	i := FileExist(filePath) ? FileOpen(filePath, 0) : FileOpen(filePath,3),text := i.Read(),i.Close(),o := FileOpen(filePath, 5),o.WriteLine(data),o.Write(text),o.Read(0),o.Close()
}

FileAppend(filePath, data:="",keepOpen:=false){	;keepOpen best used with rapid logging
	static
	( IsObject(o) ? o.write(data) & o.Read(0) : (o := FileOpen(filePath,"a")) & o.write(data) & o.Read(0) ),( keepOpen ? "" : o.close() & (o:="") )
}

FileRead(filePath){
	i := FileOpen(filePath,0).Read(), i.Close()
	Return i
}

Vi Veri Veniversum Vivus Vici

Return to “Tips and Tricks”

Who is online

Users browsing this forum: No registered users and 2 guests