Absurd behavior when reading from STDIN

[Guest]

Quite a simple situation. I have two files:
send.ahk writes strings to STDOUT every 1 second
receive.ahk reads strings from STDIN as soon as they arrive and then writes them to STDOUT

Then I pipe everything together to get the output in the same console, like this:
AutoHotkeyU32.exe send.ahk | AutoHotkeyU32.exe receive.ahk | more

send.ahk

Code: Select all

STDOUT := FileOpen("*", "w")

loop 10 {
	msg := "Hello " a_index
	STDOUT.WriteUInt(strlen(msg))
	STDOUT.Write(msg)
	STDOUT.Read(0) ;force buffer flush
	sleep 1000
}

receive.ahk

Code: Select all

STDIN := FileOpen("*", "r")
STDOUT := FileOpen("*", "w")

loop 50 {
	msgLen := STDIN.ReadUInt()
	STDOUT.Write( STDIN.Read(msgLen) "`n" )
	STDOUT.Read(0) ;force buffer flush
}

The output I would expect from all this is:

Code: Select all

Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 6
Hello 7
Hello 8
Hello 9
Hello 10

But what happens instead is this:

Code: Select all

Hello 1
Hello 1
Hello 2
Hello 1
Hello 2
Hello 3
Hello 1
Hello 2
Hello 3
Hello 4
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 6
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 6
Hello 7
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 6
Hello 7
Hello 8
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5
Hello 6
Hello 7
Hello 8
Hello 9
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5

I'm clueless as to what in the world is happening here, or how to solve it.
Anybody knows what's causing this strange behavior?

I'm using AHK 1.1.24.02 under Windows Vista x86

[lexikos]

It's a bug. If a prior File.Read() has read exactly all of the data from the buffer, File.ReadNum() causes the buffered data to be reused.

It doesn't happen if File.ReadNum() reads to the end of the buffer, because in that case it resets the buffer.

It doesn't happen if you use only File.Read().

It is much less likely to happen with normal files because you generally only reach the end of the buffer every 8KB or at the end of the file.

You can work around it:

• by using File.RawRead() in combination with NumGet()/StrGet() instead of File.ReadUInt() and File.Read().
• by using a fixed-length numeric string instead of a raw number for the message length; i.e. use only File.Write() and File.Read().

There are also less optimal ways, like replacing File.Read() with File.ReadChar() in a loop. Resetting the read buffer with File.__Handle or File.Write("") would seem to solve the problem, but if multiple messages are waiting in the pipe, the first would be read and the rest discarded.

[Getfree]

Thanks lexikos.
Quite a strange bug.

It works using only .RawRead() as you suggest. However, one good thing about .ReadNum() is that it doesn't return if there's nothing to read (it blocks waiting for input).
But since I can't use that method now, I don't have a way to wait.
.RawRead() doesn't wait, it just returns zero.
A workaround could be to keep polling the stream endlessly, but that doesn't seem quite right.

Do you know of another way to wait for input in the stream rather than polling for it?

[lexikos]

.RawRead() doesn't wait, it just returns zero.

Not true. All of the Read functions are blocking. They all wait until some data is available. (Rather, if the buffer is empty they will indirectly call the Windows ReadFile function, which waits for data.)

RawRead won't guarantee a minimum number of bytes, but neither does ReadUInt. It's just very unlikely for a read request to return less than 4 bytes (unless the other end wrote something unexpected, like 1 character).

In this case, where the messages are short, it's very unlikely that you'll get an incomplete read once any of the data is available (i.e. anything less than the entire message). In some cases you might need to loop RawRead to get the required number of bytes. RawRead only performs a single read request (or none if the buffer already contains enough to fill your variable).

[Getfree]

You are right. I got confused with the mixed results from that bug.
Thanks for the detailed explanation.

I noticed that when .RawRead() is waiting, it block the whole AHK process. You can't fire hotkeys or menues while it's waiting for input.

I solved the problem creating an OS thread with:
DllCall("CreateThread", ptr,0, uInt,0, ptr,RegisterCallback("function_name","F"), ptr,0, uInt,0, ptr,0, ptr)

But I wonder if there's some way of achieving the same with AHK functionality alone, without resorting to DLL calls.
Is there?

[lexikos]

DllCall("CreateThread", ptr,0, uInt,0, ptr,RegisterCallback("function_name","F"), ptr,0, uInt,0, ptr,0, ptr)

Don't do it, unless you like HEADACHES and debugging problems that seem to make no sense. AutoHotkey is not designed to run on multiple threads. You may find more details by searching for other forum topics about multi-threading.

To work around the problem you can do one of:

• Use the Windows file I/O functions directly to do "overlapped" I/O. (Does not work with console buffers, but probably works with pipes.)
• Use a separate program or another language.
• Run one script to do the background stuff (e.g. file I/O) and a separate script to interface with the user (hotkeys, menus, GUIs, etc.). Do this either with separate script processes or with AutoHotkey.dll/AutoHotkey_H.