14 replies to this topic

[stevenp]

Instead of using FileAppend(), decided to try the File object and the Benchmark shows that it's 5-10x times faster.
However, there's an issue. Commonly right after a FileAppend() is performed, the text file is ready for Read-Write-Delete operations in some File Manager or a Text Editor.
But that's not the case when the File is Opened by the Script.

file := FileOpen("C:\Test.txt", 2|0x100|0x200|0x400) ; Append, Shared

after performing a file.Write on a non-existing file, the file size is 0 bytes, if the file exist, the appended data is not visible in the file until a file.Close is performed or the Script is Reloaded/Closed.

Are there other "Sharing mode flags" or is this a normal behavior?
Is there a method to Update the file while it's Open by ahk, so that the appended text will be visible immediately after a file.Write?

[Lexikos]

Read/write is buffered for performance. I wouldn't recommend keeping the file open for long, but if you must, there are a few ways to force the data to be written to disk. This is the simplest:

file.__Handle

Also, I recommend using the string options. It is less error-prone, and makes the script more readable.

file := FileOpen("C:\Test.txt", "a")  ; Sharing is enabled by default, -rwd to override.

[stevenp]

Thanks a lot, works like a charm!
Just benchmarked with/without .__Handle and there's no slowdown.
Why it's not enabled by default?

[Rseding91]

Thanks a lot, works like a charm!
Just benchmarked with/without .__Handle and there's no slowdown.
Why it's not enabled by default?

You're probably doing very small reads/writes which windows its self might be buffering or the HD might be buffering. You're computer is probably also decent so the HD can keep up with the read/writes. Not all HDs are the same and they arn't always under none/light loads. At heavy load write buffering could be the difference between a .03 MS write operation or a .5 second write operation.

There are just to many variables to say disabling buffering makes no difference.

[stevenp]

it's SSD HDD

[stevenp]

I've found that when file.__Handle is used about 20-50 times per second, there's a 300-500 ms delay occurs after 5-30 second of usages.
That's unacceptable when an application is driven with hotkeys.
Will change subroutines so that files will be closed when thread is finish running.

Lexikos:
there are a few ways to force the data to be written to disk

Lexikos, what other ways can you offer?

[stevenp]

Currently I've decided to update/refresh the files from within a timer

SetTimer RefreshFilesTimer, 500
RefreshFilesTimer:
if IsObject(fileLog) && (A_TimeIdlePhysical > 500) && (A_TimeIdlePhysical < 1000) {
	file1.__Handle
	file2.__Handle
	...
}

Do you know a way to determine whether a file is already refreshed/updated/written?

[Lexikos]

Lexikos, what other ways can you offer?

It would probably be a waste of time to explain them, as they all internally do the same thing.

there's a 300-500 ms delay occurs after 5-30 second of usages.

So don't do it.

Do you know a way to determine whether a file is already refreshed/updated/written?

Read the file. That would probably make it pointless, though.

If there's no data in the buffer, nothing will be written to disk.

[stevenp]

a little benchmark of interchanging data between scripts from within a Function()

File Write vs. Messaging
Send_WM_COPYDATA			 20 ms
Post/SendMessage, AHK_MSG 10 ms
FileAppend					 1.5 ms
file.Write  file.Close	 1   ms
file.Write  file.__Handle 0.05 ms (occasionally lags)
file.Write					 0.03 ms
return (no code)			 0.001 ms

[stevenp]

Currently there are two solutions:
* Refresh Timer
* Refresh when a thread finishes
There's a OnExit label available, but a OnReturn or OnThreadExit label is missing and it's very welcome and probably would be useful in other cases too. Lexikos, could you implement it?

[Lexikos]

If the goal is for two scripts to communicate, writing to a file is probably the worst method. You cannot compare messaging (which is a complete solution), to file writing (which is only half of a solution; the other half being detecting the change and reading the file).

could you implement it?

No.

[just me]

Do you know a way to determine whether a file is already refreshed/updated/written?

Within the writing script you may check File.Pos against File.Length. Whenever File.Pos exceeds File.Length data are buffered internally.

[Lexikos]

That's a bug. If you rely on it, your script will break when it is fixed.

[just me]

Will the fix change anything else than File.Length after File.Write() (row 2 in the ListView of the following example)?

#NoEnv
FilePath := A_ScriptDir . "\Test.txt"
ToWrite := ""
Loop, 512
   ToWrite .= "A"
FO := {}
FileDelete, %FilePath%
HFile := FileOpen(FilePath, "a")
FO.Insert({Pos: HFile.Pos, Len: HFile.Length, Cmd: "FileOpen"})
BW := HFile.Write(ToWrite)
FO.Insert({Pos: HFile.Pos, Len: HFile.Length, Cmd: "File.Write()"})
HFile.__Handle
FO.Insert({Pos: HFile.Pos, Len: HFile.Length, Cmd: "File.__Handle"})
FileGetSize, S1, %FilePath%
HFile.Length := HFile.Pos
FO.Insert({Pos: HFile.Pos, Len: HFile.Length, Cmd: "File.Length := File.Pos"})
FileGetSize, S2, %FilePath%
HFile.Close()
Gui, Color, F0F0F0
Gui, Margin, 20, 20
Gui, Add, ListView, w400 r5 Grid, File.Pos|File.Length|After
For K, O In FO
   LV_Add("", O.Pos, O.Len, O.Cmd)
Loop, % FO.MaxIndex()
   LV_ModifyCol(A_Index, "AutoHdr")
Gui, Add, Text, xm  Section, FileGetSize after File.__Handle
Gui, Add, Text, xs y+5, FileGetSize after File.Length := File.Pos
Gui, Add, Text, ys cRed, % SubStr("00000" . S1, 1 - StrLen(BW))
Gui, Add, Text, y+5 cBlue, % SubStr("00000" . S2, 1 - StrLen(BW))
Gui, Show, , File - bytes written = %BW%
Return
GuiClose:
GuiEscape:
ExitApp

[stevenp]

Nice GUI for testing file.__Handle. Thanks!
HFile.Length returns 0 bytes before HFile.__Handle method and then returns 512 b after it is performed.
And FileGetSize() shows 0 bytes until HFile.Length := HFile.Pos is performed
Thanks just me for the great tip! Now I could use the following code in the timer:

if IsObject(file) { 
		if (file.Length < file.Pos)
			file.__Handle
	}

but, a file.Length < file.Pos test takes around 0.01 - 0.015 ms for each file object
while when only file.__Handle is used, it takes almost nothing - 0.001 - 0.002 ms

Lexikos:
You cannot compare messaging (which is a complete solution), to file writing

I've benchmarked messaging, because I wanted to use another instance of ahk (as ahk interpreter don't know about multithreading) for performing FileAppend, which takes 1.5 ms. The purpose of this 2nd Instance would be to only Write Text to Files. But Messaging is 100-500x times slower than File.write:

average 10 ms Post/SendMessage (depends on CPU?)
vs.
     average 0.03 ms file.Write (depends on HDD IOPS)

When Logging/Debugging needed without affecting script performance, the file.Write with it's 0.03 ms delay is more than acceptable.
And the 20 ms delay when using WM_COPYDATA is simply unacceptable.
Messaging is really-really too slow when sending data between AHK scripts, it have completely different purpose.
So, for transparency, better to write data to file within the main script and Closing the file only when it's necessary.

Still, a OnThreadReturn label is very welcome. Lexikos, why you cannot implement it?