Why you shouldn't use FileAppend to continuously write to the disk

Put simple Tips and Tricks that are not entire Tutorials in this forum
DRocks
Posts: 565
Joined: 08 May 2018, 10:20

Re: Why you shouldn't use FileAppend

Post by DRocks » 10 Feb 2019, 09:31

Ok cool. Thanks for taking time for such a clear answer.
I understand now. My current needs are always with small file sizes so I admit never tinking about the size problem with memory.

That loop example is quite interesting.
So for what I understand, small filesize would be totally fine for using just one variable and 1 file write. But as soon as you deal with bigger content its wise to care about memory loads.

Meroveus
Posts: 44
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

Post by Meroveus » 11 Feb 2019, 00:48

As I mentioned on Discord (username there is maugr1s) I tried to use the technique where you open the file at the beginning, write several times throughout the script, then close it at the end.

When I do this, the file shows in the directory as 0 length and cannot be read by an external program until it is closed.
I opened the file with the "a" flag

I tested a crash by using pskill to kill all instances of autohotkey.exe - which bombs the running script out of existence.
Under such circumstances, the buffer is not flushed, and the contents of the log are lost.

However, in the examples for the FileOpen() function is this piece of code:

Code: Select all

; Open a console window for this demonstration:
DllCall("AllocConsole")
; Open the application's stdin/stdout streams in newline-translated mode.
stdin  := FileOpen("*", "r `n")  ; Requires [v1.1.17+]
stdout := FileOpen("*", "w `n")
; For older versions:
;   stdin  := FileOpen(DllCall("GetStdHandle", "int", -10, "ptr"), "h `n")
;   stdout := FileOpen(DllCall("GetStdHandle", "int", -11, "ptr"), "h `n")
stdout.Write("Enter your query.`n\> ")
stdout.Read(0) ; Flush the write buffer.
query := RTrim(stdin.ReadLine(), "`n")
stdout.WriteLine("Your query was '" query "'. Have a nice day.")
stdout.Read(0) ; Flush the write buffer.
Sleep 5000
When I include the line:

Code: Select all

logFile.Read(0)
after the writes, the buffer is flushed, and the crashes don't lose data.

I would rather lose performance than data.

Thanks for your help!

User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend

Post by nnnik » 11 Feb 2019, 01:06

Yeah that is a problem - I wanted to make a few tests myself but got side-tracked at the end. From what I understood from the docs a, w, rw and r includes share flags so the locking behavior seems rather strange to me. Which system are you on?
Edit: Now I tested it myself and the buffer is indeed not flushed when I crash the script - I guess this is to prevent the program from writing bad data to the file - I need to change some stuff in the topic.
When Opening with "w" mode I was able to read the data from another program while it was still opened in AutoHotkey - same goes for the "a" mode.
Here is the script I used:

Code: Select all

file := FileOpen("crash.log", "w")
file2 := FileOpen("crash2.log", "w")

Loop {
	file.writeLine(A_Index)
	file2.writeLine(A_Index)
	file2.read(0)
	if (randomChance(1/100000)) {
		Msgbox about to crash
		causeCrash()
	}
}

randomChance(chance := 0.5) {
	Random,  randThrow,  0.0, 1.0
	return (randThrow<chance)
}

causeCrash() {
	causeCrash()
}
Recommends AHK Studio

Meroveus
Posts: 44
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

Post by Meroveus » 11 Feb 2019, 12:55

I'm on Windows 10 1803 64 bit.
I modified this somewhat by using:

Code: Select all

logFile := FileOpen(logPath,"a-d",CP1252)
to open the file.
I need it to append, I could use rw-d and position the pointer to the end, but it makes no difference
The -d prevents the file being deleted while logging. Seems prudent.
I use the alternate code page for convenience -- elsewhere the script uses UTF-8

Using the line

Code: Select all

logFile.Read(0)
To flush the cache seems reliable enough. In several tests the data survived intact.
Perhaps this could be documented in the section for the Read Method.

What method do you use to deliberately crash the script?

Meroveus
Posts: 44
Joined: 23 May 2016, 17:38

Re: Why you shouldn't use FileAppend

Post by Meroveus » 11 Feb 2019, 17:43

Meroveus wrote:
11 Feb 2019, 12:55
What method do you use to deliberately crash the script?
Ha! Slick -- I just figured it out. I thought CauseCrash() might be in a library or something.

So the take-aways are:
  • Use file objects rather than file commands
  • Open once, write many, close once (on exit)
  • use Read(0) method following writes to flush buffer to prevent data loss.
  • lock file for deletion to prevent data lost due to inadvertent deletion while executing
All the best...
--M

User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by nnnik » 19 Feb 2019, 05:40

I have included the newly gathered information.
Thanks for all the intereting and nice replies :)
Recommends AHK Studio

0x00
Posts: 89
Joined: 22 Jan 2019, 13:12

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by 0x00 » 23 Feb 2019, 00:22

Thanks for the insight, I've resolved to not use FileAppend as much...

FileAppend() & FilePrepend() incorporating your insights...

Code: Select all


Loop 10{	;file is only prepended to once & appended to the remainder of the loop,as FileAppend keeps the file open.
	FilePrepend("D:\Ut.txt", A_Now)
	FileAppend("D:\Ut.txt", "`n" A_Now " - Appended",true)	;keeps file open & also prevents other writes to file...
}
FileAppend("D:\Ut.txt")	;release file


Loop 10{	;file is both prepended & appended to
	FilePrepend("D:\Ut.txt", A_Now "loop2")
	FileAppend("D:\Ut.txt", "`n" A_Now " - Appended")
}

run, D:\Ut.txt
exitapp

FilePrepend(filePath, data){
	i := FileExist(filePath) ? FileOpen(filePath, 0) : FileOpen(filePath,3),text := i.Read(),i.Close(),o := FileOpen(filePath, 5),o.WriteLine(data),o.Write(text),o.Read(0),o.Close()
}

FileAppend(filePath, data:="",keepOpen:=false){	;keepOpen best used with rapid logging
	static
	( IsObject(o) ? o.write(data) & o.Read(0) : (o := FileOpen(filePath,"a")) & o.write(data) & o.Read(0) ),( keepOpen ? "" : o.close() & (o:="") )
}

FileRead(filePath){
	i := FileOpen(filePath,0).Read(), i.Close()
	Return i
}


Sam_
Posts: 146
Joined: 20 Mar 2014, 20:24

Re: Why you shouldn't use FileAppend

Post by Sam_ » 28 May 2019, 16:59

Meroveus wrote:
11 Feb 2019, 17:43
Meroveus wrote:
11 Feb 2019, 12:55
What method do you use to deliberately crash the script?
Ha! Slick -- I just figured it out. I thought CauseCrash() might be in a library or something.
This is what came to mind:

Code: Select all

;;; !!! Do not run this unless you want to crash the script !!! ;;;
#Persistent
SetTimer, CauseCrash, -1000 ; Will trigger a crash after a few sec.
Return


CauseCrash(){
	CauseCrash() ; Gives ErrorCode 3221225725 (0xC00000FD) - Stack Overflow?
}
My second thought was go for Access Violation:

Code: Select all

CauseCrash(){
	Loop
		NumGet(0,A_Index) ; Gives ErrorCode 3221225477 (‭C0000005‬) - Access Violation
}

lexikos
Posts: 9554
Joined: 30 Sep 2013, 04:07
Contact:

Re: Why you shouldn't use FileAppend

Post by lexikos » 31 May 2019, 05:43

The OP contains significant exaggeration and some false assumptions.
nnnik wrote:
07 Feb 2019, 08:35
Closing the file will force windows to completely flush the buffer onto the disk before execution continues.
Microsoft's documentation explicitly states otherwise:
If there is data in the internal buffer when the file is closed, the operating system does not automatically write the contents of the buffer to the disk before closing the file.
Source: Flushing System-Buffered I/O Data to Disk - Windows applications | Microsoft Docs
Closing the File object will cause the File object to pass the contents of its write buffer to WriteFile and then call CloseHandle. It does not force anyone to write the contents to disk (even if the file is on a disk).

The File object and FileAppend both use an internal class named TextStream. TextStream currently has an 8KB buffer that it uses for both reads and writes, but for simplicity, never uses it for both purposes at the same time. This is why calling Read(0) flushes the write buffer. The buffer is also flushed if you call File.__Handle (or in v2, File.Handle), so that the position of the file pointer and actual content of the file (as seen by whatever Win32 function you're calling) will be what you expect. However, this only synchronizes the File object's perspective with the Win32 one; it does not guarantee that data is written to physical storage.

When you read data from file, the file pointer is advanced by as many bytes as were read. When you write data to file, it is written at the position indicated by the file pointer, which is then moved by as many bytes as were written. So in effect, writing n bytes will cause the next n bytes (if there are that many) to be overwritten and therefore not be returned by the next read. Reads and writes could share the buffer by emulating this behaviour. Doing so would improve the performance of scripts that intersperse reads and writes within an 8KB block. In that case, Read(0) would have no reason to flush the write buffer, so it's probably better to use File.Handle if you want to flush the buffers.

The File object performs buffering because calls to ReadFile and WriteFile are costly. I believe this is mostly because of the overhead of switching to kernel mode (and perhaps calling the file system driver), not because of disk write performance. Originally when Unicode support was added, ReadFile and WriteFile were called very frequently (perhaps for every character). When I tested the performance, I found that the overhead of these calls dwarfed any differences caused by storage devices (such as between a hard disk drive, SSD or RAM disk).

nnnik wrote:
11 Feb 2019, 01:06
Edit: Now I tested it myself and the buffer is indeed not flushed when I crash the script - I guess this is to prevent the program from writing bad data to the file - I need to change some stuff in the topic.
The buffer you are flushing is in the TextStream object (within the File object), which the OS knows nothing about. The buffer is not deliberately discarded; it is lost because the program has crashed and no further program logic is executed. It is like accumulating your data in a variable with the intention of passing it to FileAppend, but then having the program crash before FileAppend is called.

nnnik wrote:
07 Feb 2019, 08:35
Each time you close a file Windows will back up this new version of the file in addition to removing the old one.
I think you have a mistaken idea about how shadow copies work. Shadow copies are taken on request, such as when backup software runs or a system restore point is created, not whenever you write to a file. Repeatedly opening and closing a file will not cause multiple backups of the file. Backups of the user's files might never be (and generally are not) created via shadow copy without the use of additional software (or configuring File History).

As for your assumption that an antivirus would wait until the file is closed, I think it is unfounded. I would think that most antivirus programs are like a black box to anyone but the developer, and each one may be different; perhaps some behave the way you say, and some do not.


Expressions have overhead, and objects have more overhead. In some cases the overhead of FileOpen + File.Write can outweigh any performance gain received from buffering and keeping the file open, and it is probably always slower than accumulating the data in a variable and writing once with FileAppend. Whether it helps or hinders performance should be confirmed by benchmarking your script, if that kind of performance matters to you. But if you can't see the difference without benchmarks, it probably doesn't matter in the real world.


It appears that all of the arguments against FileAppend so far are based on the assumption that it opens and closes the file each time, but consider Loop Read's OutputFile parameter. In that case, the first call to FileAppend which omits the Filename opens the output file, and it is closed only when the loop stops. Each call to FileAppend (omitting the Filename) utilizes the same buffering as File.Write.

jeeswg wrote:
07 Feb 2019, 15:09
But is it possible to reserve file space for the text? Otherwise the file appending via the File object might be inefficient.
How does this relate specifically to the File object and not FileAppend? You could accumulate data into a variable and then pass it to File.Write once, or you could call FileAppend repeatedly.

In fact, you can "reserve file space" by setting File.Length, but there is probably no guarantee that the OS will physically reserve space on the disk, or that it will do so efficiently, or that the reserved space will be contiguous (it could be fragmented).

File.Write typically only writes when the 8KB buffer is near full, whereas FileAppend writes immediately; but even so, the file system has a minimum allocation unit (often 4KB) which would reduce the chance of small writes causing fragmentation.

swagfag wrote:
07 Feb 2019, 17:32
FileOpen ultimately calls WriteFile. can it be altered to expose nNumberOfBytesToWrite?
1. FileOpen corresponds to CreateFile, not WriteFile. It just opens the file handle.
2. When you write text, you don't want to specify the number of bytes. You want exactly that text encoded into bytes and written, and that's what the File object's methods do, if you ignore buffering.
3. RawWrite accepts a byte count.

If you want data flushed from the buffer immediately by each Write/RawWrite call, what you need is an option for that, not to expose some parameter of WriteFile. (This reminds me of the XY problem.)

User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by jeeswg » 31 May 2019, 07:22

The thread title implies 4 choices:
1) FileAppend continuously [obviously slow because of repeated opening/closing]
2) File.Write continuously [how much slower is continuous v. once only, can 'reserving file space' via File.Length make it more efficient?]
3) FileAppend once-only [once-only should be faster, however, disadvantage: you'd have to store a massive file in memory]
4) File.Write once-only [equivalent to, but slightly more verbose than 3]

The only interesting comparison is 2 v. 3. Here is some potential code for benchmark tests:

Code: Select all

;File.Length can be used in a similar way to VarSetCapacity,
;to prepare a certain amount of space in the file,
;but it sets the *size*, not the *capacity*,
;so it should be subsequently reduced, if not all of the space was needed,
;otherwise you will have extra bytes at the end of the file

;if File.Length is set, but is too small for the text,
;the File object writes beyond it,
;similarly, AHK expands variables when necessary

q:: ;append to file via File object
vText := "abcdefghijklmnopqrstuvwxyz"
vNum := 10
vPath := A_Desktop "\z temp " A_Now ".txt"
vTickCount := A_TickCount
Loop, 10000
{
	FileDelete, % vPath
	if !(oFile := FileOpen(vPath, "w`r`n"))
		return
	vSize := (StrLen(vText)+2)*vNum
	oFile.Length := vSize
	oFile.Encoding := "UTF-16"
	oFile.Write(Chr(0xFEFF)) ;BOM
	Loop, % vNum
		oFile.WriteLine(vText)
	;MsgBox, % oFile.Pos
	oFile.Length := oFile.Pos
	oFile.Close()
	;FileGetSize, vSize, % vPath
}
;MsgBox, % vSize
;Run, % vPath
MsgBox, % Clipboard := A_TickCount - vTickCount
return
;results: 16255 15101 15288
;results: 13182 13369 14212 ;'oFile.Length := vSize' commented out

w:: ;append to variable, write variable to file via FileAppend
vText := "abcdefghijklmnopqrstuvwxyz"
vNum := 10
vPath := A_Desktop "\z temp " A_Now ".txt"
vTickCount := A_TickCount
Loop, 10000
{
	FileDelete, % vPath
	vSize := (StrLen(vText)+2)*vNum
	vOutput := ""
	VarSetCapacity(vOutput, vSize)
	Loop, % vNum
		vOutput .= vText "`r`n"
	FileAppend, % vOutput, % "*" vPath, UTF-16
	;FileGetSize, vSize, % vPath
}
;MsgBox, % vSize
;Run, % vPath
MsgBox, % Clipboard := A_TickCount - vTickCount
return
;results: 13104 13900 12496
Using FileAppend once only appeared to be a little faster than using File.Write continuously. Curiously, using File.Length in advance, to set the size, appeared to slow things down.

Btw does File.Length create a sparse file. I.e. if you don't fill up the space, are the remaining bytes assumed to be null bytes, but not explicitly stored.
How to generate a large sparse file natively only with ahk? - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=26633

Regarding flushing, I have a query here, re. the IniWrite source code.
IniWrite: flush WritePrivateProfileString - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=59262
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA

User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by nnnik » 31 May 2019, 13:15

I think you have a mistaken idea about how shadow copies work. Shadow copies are taken on request, such as when backup software runs or a system restore point is created, not whenever you write to a file. Repeatedly opening and closing a file will not cause multiple backups of the file. Backups of the user's files might never be (and generally are not) created via shadow copy without the use of additional software (or configuring File History).
I will correct that. What I intend to say is that it is possible that such programs are installed/such settings activated.
As for your assumption that an antivirus would wait until the file is closed, I think it is unfounded.
My assumption is that both the anti-virus and the file history/shadow copy program will be reasonably programmed or reasonably optimized.
If they are not and cause performance issues that easily make programs, that e.g. create file logs, unusuable they are at fault and its safe to ignore these anti-virus/shadow copy programs.
One such optimization is waiting until a program is finished with a file and then handle it or handle what you can handle while its being written but expect it to change.
In both cases its important that we communicate our intentions. Opening when you start writing and closing when you are done is the most basic step to that.
Expressions have overhead, and objects have more overhead. In some cases the overhead of FileOpen + File.Write can outweigh any performance gain received from buffering and keeping the file open, and it is probably always slower than accumulating the data in a variable and writing once with FileAppend. Whether it helps or hinders performance should be confirmed by benchmarking your script, if that kind of performance matters to you. But if you can't see the difference without benchmarks, it probably doesn't matter in the real world.
I think the same. Acumulating the data first in a variable will be faster. But it will also keep it all in RAM. Like I stated in my original post you need to balance between using memory and writing to disk.
I personally don't have a lot of free RAM to give. If I see your script/program using too much I'll toss it away and look for an alternative or simply not use it. For background programs - that AutoHotkey is commonly used for - thats especially important.
You need to be able to keep them running in the background. If your script doesn't write large amount of data to the disk the performance doesn't matter either way.
I can safely use .write both for small and for large files without making my user run into major headaches. That's why I reccomend it at all times. Your code is good and you don't have to overthink it.
The File object and FileAppend both use an internal class named TextStream. TextStream currently has an 8KB buffer that it uses for both reads and writes, but for simplicity, never uses it for both purposes at the same time. This is why calling Read(0) flushes the write buffer. The buffer is also flushed if you call File.__Handle (or in v2, File.Handle), so that the position of the file pointer and actual content of the file (as seen by whatever Win32 function you're calling) will be what you expect. However, this only synchronizes the File object's perspective with the Win32 one; it does not guarantee that data is written to physical storage.

When you read data from file, the file pointer is advanced by as many bytes as were read. When you write data to file, it is written at the position indicated by the file pointer, which is then moved by as many bytes as were written. So in effect, writing n bytes will cause the next n bytes (if there are that many) to be overwritten and therefore not be returned by the next read. Reads and writes could share the buffer by emulating this behaviour. Doing so would improve the performance of scripts that intersperse reads and writes within an 8KB block. In that case, Read(0) would have no reason to flush the write buffer, so it's probably better to use File.Handle if you want to flush the buffers.
That's unfortunate.
As far as I am concerned the use of the file-object should ne limited to a stream-interface that allows the read and write of data-streams.
The only difference I can tell from the top of my head from the direct file-object is the handle property.
The polymorphic stream-interface should not assume that it has a handle that is corresponding to a very specific win32 file handle meaning.
As it stands there is no valid method to flush an object compatible with the stream-interface. I'd have to add a new method for flushing that (at least in v1 - didnt keep up with the in-depth changes of v2 lately) makes the built-in fileObject incompatible with the interface.
I could also add a handle property to the stream-interface and make it flush but return 0 or similar but that would be a very hackish solution.

Would you accept a pullrequest that adds a flush method to the file object that fulfills this task?


Thanks for the in-depth explination. I will try to add the new info to the original post and correct my assumptions.
Recommends AHK Studio

lexikos
Posts: 9554
Joined: 30 Sep 2013, 04:07
Contact:

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by lexikos » 31 May 2019, 22:29

nnnik wrote:
31 May 2019, 13:15
What I intend to say is that it is possible that such programs are installed/such settings activated.
I'm not sure you got my point. Shadow copies are irrelevant. Even if you have a backup program installed or File History activated, it will only create shadow copies when the backup runs, which will happen on a schedule or whenever the user requests it, not every time you write to a file. See Wikipedia; "users cannot trigger the creation of new versions of an individual file, only the entire volume".
My assumption is that both the anti-virus and the file history/shadow copy program will be reasonably programmed or reasonably optimized.
You've made further assumptions about what "reasonably programmed" and "reasonably optimized" mean for antivirus software, or specifically that it would include waiting until the file is closed before scanning it. Have you ever developed an antivirus, worked on optimizing one, or come across a reputable source confirming your theories?

I haven't, but I've worked on many computers with a variety of antivirus programs (using, repairing or troubleshooting issues, sometimes caused by the antivirus). In my experience, they are all imperfect (like every other program). All of them have bugs and sometimes cause performance issues. As a matter of fact, some well known antivirus programs have been known to cause issues with past versions of Ahk2Exe, before the file is closed.
If they are not and cause performance issues that easily make programs, that e.g. create file logs, unusuable they are at fault and its safe to ignore these anti-virus/shadow copy programs.
I could say the same about log files that are written with FileAppend or any similar technique. There is little reason for an antivirus to perceptibly slow down access to a text file.
But it will also keep it all in RAM.
RAM is there to be used. It's all relative; "a lot", "too much", etc. I think it's typical for scripts to be processing text. Even on older systems, the default #MaxMem (64MB) is usually enough, and won't tax the system overly unless it's constantly near the limit, which would be a problem regardless of what the script does. If I'm building a string to write to file, it is generally small enough that RAM usage is a total non-issue.

It is possible that if the RAM is available, the data you carefully write to the file in small increments actually accumulates in disk cache, in RAM, before it is all written to disk. But instead of just passing it all at once, you have made several WriteFile calls, incurring overhead that could have been avoided.

It is also possible that the total data is less than 8KB, so by using the File object you have reserved more RAM than if you were to build the string in a variable.

It is also possible that in some cases, keeping the file open provides some real world benefit.

For me, other concerns are more significant; even just whatever is more convenient to write, or suits my own programming style.
For background programs - that AutoHotkey is commonly used for - thats especially important.
Background scripts that constantly process large volumes of data? I don't think that's common.
If your script doesn't write large amount of data to the disk the performance doesn't matter either way.
Maybe that's what I'm saying; it doesn't matter either way.
I can safely use .write both for small and for large files without making my user run into major headaches.
The same is true for FileAppend.
The polymorphic stream-interface should not assume that it has a handle that is corresponding to a very specific win32 file handle meaning.
If you're referring to the File object's complete interface, that is not a "polymorphic stream-interface" - it's a File object. If you're referring to some theoretical subset of the methods suitable for generic streams, clearly that doesn't include a Handle property or assume that one would be present. The current implementation has no provision for flushing buffers; that's just a side-effect of Read(0) and Handle. The original object did not have buffering.

Actually, I mentioned the TextStream class; it does not have a file handle. The handle belongs to the derived TextFile class, which is what the File object actually uses.
Would you accept a pullrequest that adds a flush method to the file object that fulfills this task?
As with any pull request, I may or may not accept it, depending mostly on the content of the pull request. I do not object to the idea of a flush method.

User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by nnnik » 01 Jun 2019, 02:54

The same is true for FileAppend.
The whole topic was spawned by the abysmal performance of FileAppend being used in a loop to write a lot of data.
A lot of data being more than 300MB in this case. The task was reading multiple database files and finding and removing double entries that happened due to a faulty device (afaict).
With FileAppend in the loop the program took a lot longer to complete. I don't remember the actual number but I think the difference of calling FileAppend in the loop vs. calling File.write in a loop.
This could still have been dealt with inside the RAM for one of those database files - but the script had to reate many.

The general reccomendation for file.write I hand out greatly simplifies the context and circumstances.
But it also makes sure that the worst case performance is better than the example I gave above.
Of course you can create faster programs when you know what you are doing.
Of course you can program however you want and as lazy as you want.
But when the users ever run into the issue of FileAppend in a loop being slow they might remember this topic and might be able to help themselves.

We can probably write entire essays about the assumptions one should make when dealing with AV or FileSynchronisation/Shadow copy programs.
(I know for a fact that dropbox starts synchronising as fast as it can)
And they probably arent even going to be as optimized as I mentioned before. However its still right to express what your program does correctly.
Using fileAppend in a loop is less expressive than opening and closing the file when you start and end writing.

Yes I am refering to a theoretical subset. The name I use is Stream Interface (Data Stream and Interface from the context of java programming)
I might eventually make a Pull Request.
Recommends AHK Studio

conracer
Posts: 6
Joined: 26 Jul 2020, 11:51

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by conracer » 08 Nov 2020, 17:50

If you delete the file and do FileAppend afterwards, you are also deleting the files permissions, for example. Here is a better solution:

Code: Select all

FILEPATH=%A_AppData%\vlc\vlcrc
if(!FileExist(FILEPATH)) {
	MsgBox Couldn't find file.
	return
}
file:=FileOpen(FILEPATH,"rw")
if(!IsObject(file)) {
	MsgBox Can't open "%FILEPATH%" for writing.
	return
}
file.Seek(0) ; In this case, the original file has a "Byte Order Mark"
FileContents:=file.Read()
ReplaceText:="hotkeys-y-wheel-mode=-1" ; Deactivate »Volume control with mouse wheel«
FileContentsNew:=StrReplace(FileContents,"#hotkeys-y-wheel-mode=0",ReplaceText)
file.Length:=0 ; Otherwise it would leave the rest of the file as it was. Imagine it like typing text with »insert«-button activated.
file.Write(FileContentsNew)
file.Close()
MsgBox % "Configuraton has been written to """ FILEPATH """." (FileContents=FileContentsNew?"`n`nAlready existing setting:`n" ReplaceText:"")

User avatar
lmstearn
Posts: 688
Joined: 11 Aug 2016, 02:32
Contact:

Re: Why you shouldn't use FileAppend to continuously write to the disk

Post by lmstearn » 17 Nov 2021, 22:37

Interesting. One could possibly update the backup file version by editing the file, then initiating a differential backup using File Level Schema Support. The exercise looks rather involved, and is probably best performed in kernel mode.
:arrow: itros "ylbbub eht tuO kaerB" a ni kcuts m'I pleH

Post Reply

Return to “Tips and Tricks (v1)”