Get StdOut From Process Thread

[Masonjar13]

I've been working on a KF2 manager (yes, it is a game, but the concept I believe may be universal). Initially I tried StdOutStream() on an alternate thread, which works as well as %comspec% KFServer.exe >> stdOut.txt. Testing will require downloading the server files (you may use my installer and delete the folder afterwards). What appears to be the issue is that there are two separate threads contributing separate StdOut streams to the same window. The first is used to display possible errors prior to loading the server. The second is the output of the server itself, which is what I can't seem to grab. When using StdOutStream(), the pre-loading errors/warnings are caught by the callback, but the output of the server doesn't get caught by the callback. While piping out via CMD prompt, it catches the pre-loading errors/warnings, then a second window pops up containing the server StdOut.

Thread List

The CRYPT32 thread is only present during initial loading. All others appear to be constant.

Is there a way to grab StdOut from a specific thread? Or could it be a completely different outstream type?

[qwerty12]

I think the problem is that KFServer.exe allocates a new console and then uses WriteConsole to write directly to it, bypassing any stdout redirection.

It seems the usual way to get the output in such a case appears to be "screenscraping" the console buffer (I think), see:

Why can't I redirect output from WriteConsole?
Executable called via subprocess.check_output prints on console but result is not returned

I think Lexikos might have written something along those lines in AHK: https://autohotkey.com/board/topic/3339 ... d-console/

(I'd personally inject a DLL into the process that hijacks WriteConsole using the amazing MinHook library and call it a day.)

But, for this specific program, there is an undocumented (as far as I can tell) feature: if a named pipe server is created with the name of server PID>cout, then the server will send its output there.

I had a go at writing a script that gets that output when available and writes it into a file on the desktop.

You should be aware of the following:

• The path of the KFServer.exe and the starting directory that is hardcoded in the script. Adjust the CreateProcess call as required
• The hardcoded path that the output is written to (the desktop with a name like dsaddfadfd.txt)
• The ReadFile return value (and A_LastError in case of failure) should really be checked...
• My understanding of named pipes isn't great in the slightest

This script automatically exits if the server process is closed; if you exit the script first, it will automatically kill the server (GenerateConsoleCtrlEvent doesn't seem to have any effect on KFServer, or a nicer exit would have been possible).

Code: Select all

#NoEnv
#Persistent

if (!A_IsUnicode)
	MsgBox The KF server sends strings in UTF-16. Continuing with ANSI AHK anyway (which may or may not work)... 

; Start the server:

_PROCESS_INFORMATION(pi)
VarSetCapacity(si, (siCb := A_PtrSize == 8 ? 104 : 68), 0), NumPut(siCb, si,, "UInt")

if (!DllCall("CreateProcess", "Ptr", 0
			,"Str", A_Desktop . "\KF2-Server-Installer-Manager\Installer\steamcmd\kf2server\Binaries\Win64\KFServer.exe" ; command line
			,"Ptr", 0
			,"Ptr", 0
			,"Int", False
			,"UInt", CREATE_SUSPENDED := 0x00000004 ; needs to be created suspended so that we can get KFServer's PID and to make sure the named pipe is created before KFServer tries looking for it and fails in doing so
			,"Ptr", 0
			,"Str", A_Desktop . "\KF2-Server-Installer-Manager\Installer\steamcmd\kf2server\Binaries\Win64" ; starting/working directory (,"Ptr", 0 instead to inherit this process's)
			,"Ptr", &si
			,"Ptr", &pi))
{
	DieWithLastError("CreateProcess")
}
hProcess := NumGet(pi,, "Ptr")
hThread := NumGet(pi, A_PtrSize, "Ptr")
dwProcessId := NumGet(pi, A_PtrSize * 2, "UInt")
OnExit("AtExit")

hPipe := DllCall("CreateNamedPipe"
				,"Str", Format("\\.\pipe\{:u}cout", dwProcessId)
				,"UInt", PIPE_ACCESS_INBOUND := 0x00000001 | FILE_FLAG_OVERLAPPED := 0x40000000
				,"UInt", PIPE_TYPE_BYTE := 0x00000000
				,"UInt", 1
				,"UInt", 0
				,"UInt", 0
				,"UInt", 0
				,"Ptr", 0, "Ptr")
if (hPipe == -1) {
	DieWithLastError("CreateNamedPipe")
}
; resume the process
DllCall("ResumeThread", "Ptr", hThread)
,WaitOnEvents()

WaitOnEvents() {
	global hPipe, hProcess
	static GetOverlappedResult := DllCall("GetProcAddress", "Ptr", DllCall("GetModuleHandle", "Str", "kernel32.dll", "Ptr"), "AStr", "GetOverlappedResult", "Ptr")
		  ,MsgWaitForMultipleObjectsEx := DllCall("GetProcAddress", "Ptr", DllCall("GetModuleHandle", "Str", "user32.dll", "Ptr"), "AStr", "MsgWaitForMultipleObjectsEx", "Ptr")
		  ,hEvent := DllCall("CreateEvent", "Ptr", 0, "Int", False, "Int", False, "Ptr", 0, "Ptr"), overlapped, handles, buffer
		  ,ConnectNamedPipe := DllCall("GetProcAddress", "Ptr", DllCall("GetModuleHandleW", "WStr", "kernel32.dll", "Ptr"), "AStr", "ConnectNamedPipe", "Ptr")
	if (!VarSetCapacity(overlapped)) {
		VarSetCapacity(overlapped, 32, 0)
		,NumPut(hEvent, overlapped, 2*A_PtrSize+8, "Ptr")
		
		VarSetCapacity(handles, A_PtrSize * 2) ; handles to wait on, specified in a C array for MsgWaitForMultipleObjectsEx
		,NumPut(hProcess, handles,, "Ptr")
		,NumPut(hEvent, handles, A_PtrSize, "Ptr")
		
		VarSetCapacity(buffer, 4096*2)
	}

	FileDelete, %A_Desktop%\dsffds.txt
	notConnected := !DllCall(ConnectNamedPipe, "Ptr", hPipe, "Ptr", &overlapped) && A_LastError == 997
	if (!notConnected)
		DllCall("ReadFile", "Ptr", hPipe, "Ptr", &buffer, "UInt", 4096*2, "Ptr", 0, "Ptr", &overlapped) ; if already connected, try and initially get some output
	Loop { ; stolen from Lexikos: wait on the process to terminate, while allowing messages to be pumped etc.
		r := DllCall(MsgWaitForMultipleObjectsEx, "UInt", 2, "Ptr", &handles, "UInt", 0xFFFFFFFF, "UInt", 0x4FF, "UInt", 0x6, "UInt")
		if (r == 0 || r == 0xFFFFFFFF) ; first object (process) signalled (terminated) / failure
			ExitApp
		else if (r == 1) { ; the same event is used for two purposes: the first being it's signalled when KFServer connects to it and the other is when ReadFile has something from the pipe that we should read
			if (!notConnected) {
				if (DllCall(GetOverlappedResult, "Ptr", hPipe, "Ptr", &overlapped, "UInt*", len, "Int", True)) {
					NumPut(0, buffer, len, "UShort")
					FileAppend, % StrGet(&buffer,, "UTF-16"), %A_Desktop%\dsffds.txt, UTF-16
				}
				DllCall("ReadFile", "Ptr", hPipe, "Ptr", &buffer, "UInt", 4096*2, "Ptr", 0, "Ptr", &overlapped) ; another ReadFile call is needed to get more output as it occurs
			} else {
				notConnected := False
				;DllCall("ResetEvent", "Ptr", hEvent)
				DllCall("ReadFile", "Ptr", hPipe, "Ptr", &buffer, "UInt", 4096*2, "Ptr", 0, "Ptr", &overlapped) ; start the initial reading when available
			}
		}
		Sleep -1
	}
}

AtExit()
{
	global hProcess, hThread
	OnExit(A_ThisFunc, 0)

	if (DllCall("WaitForSingleObject", "Ptr", hProcess, "UInt", 0) == 258) ; process still running
		DllCall("TerminateProcess", "Ptr", hProcess, "UInt", 1)
	DllCall("CloseHandle", "Ptr", hThread)
	DllCall("CloseHandle", "Ptr", hProcess)
	
	return 0
}

DieWithLastError(failedFuncName)
{
	dw := A_LastError
	ccherrFmt := DllCall("FormatMessage", "UInt", 0x00000100 | 0x00001000 | 0x00000200, "Ptr", 0, "UInt", dw, "UInt", 1024, "Ptr*", errFmt, "UInt", 0, "Ptr", 0, "UInt")
	MsgBox % Format("{:s} failed ({:u}){:s}", failedFuncName, dw, ccherrFmt ? ": " . StrGet(errFmt, ccherrFmt) : "")
	if (ccherrFmt)
		DllCall("LocalFree", "Ptr", errFmt, "Ptr")
	ExitApp 1
}

_PROCESS_INFORMATION(ByRef pi) {
	static piCb := A_PtrSize == 8 ? 24 : 16
	if (IsByRef(pi))
		VarSetCapacity(pi, piCb, 0)
}

It works by doing the following:

• Using CreateProcess directly to start the server exe because I need to ensure it's started in a suspended state (otherwise there will be a race-condition between the named pipe being created and the server trying to connect to it)
• The named pipe is created
• The server is resumed
• The script then enters a loop (it won't eat CPU time non-stop: MsgWaitForMultipleObjectsEx will block unless certain things happen):
  • If the process object is signalled (which happens when the server is closed), or waiting fails for some reason, then the script exits
    It waits at first for KFServer to connect to the named pipe created in the script. MsgWaitForMultipleObjectsEx will break with a return code of 2 when this happens. Here, ReadFile is called in asynchronous mode to try and get the initial output and a Boolean flag is set to say that the server is now connected
  • When there's output waiting to be read, MsgWaitForMultipleObjectsEx will break with a return code of 2 (because the same OVERLAPPED struct referencing the same Event object is used for both ConnectNamedPipe and ReadFile). Because of the toggled flag, the script will now try to read what was output and append it into the file on the desktop. ReadFile is called again to ensure the same thing happens when there's output to be read again

[Masonjar13]

Wow. Well, this will take some time for me to look through and understand (I've never heard of "named pipes" prior), but your code definitely works! I'll post back once I've got everything figured out, or if I can't figure everything out.

Do you mean this MinHook? I've never heard of it before, but that's absolutely worth my time to look into. Is there an AHK library available? I didn't see any come up with my search.

This is a huge help, thank you! And, might I ask how you figured this out, since it was (afayk) undocumented?

[qwerty12]

Do you mean this MinHook? I've never heard of it before, but that's absolutely worth my time to look into. Is there an AHK library available? I didn't see any come up with my search.

That's the one. The CodeProject page by the author describes it well. A pure AHK hooking library can be found here, but I've never had the pleasure of using it as MinHook has always worked well for me, even before I understood the basics of dllcalling in AutoHotkey.
If you Google "site:autohotkey.com MH_Initialize autohotkey" you'll find two examples of using it in AutoHotkey written by me. One example shows FindFirstFile hooked in the same AutoHotkey process so that the WinRing0 library (thanks to tmplinshi for the wonderful AHK wrapper) can find its corresponding driver (it's understandably programmed to look in the folder where AutoHotkey.exe is, but I needed it to look for the library from where my script is). The other uses HotkeyIt's InjectAhkDll - HotkeyIt's "AutoHotkey_H.dll" is loaded into Notepad.exe, which then loads MinHook which hooks lstrcmpW to get the path of an open file in a Notepad process.

A more pertinent use of it by me was actually to workaround a problem described here. I wrote a simple tray controller for ValdikSS's GoodbyeDPI. I wanted to get its output without redirecting to a file by using my usual go-to for that, but it would cause the script to hang because, as the CodeProject article says, the CRT turns off output buffer flushing.
I fired up Visual Studio, added the the MinHook NuGet package to my project and used MinHook to hijack msvcrt.dll's isatty function to always return 1 if the handle being looked at was the redirected stdout, which stops the CRT initialisation code from turning off the flushing. I then edited cyruz's function to create the GoodbyeDPI process in a suspended state and to inject my DLL into the process and resume it. Now getting GoodbyeDPI's output into the script (of which there isn't much) works fine.

This is a huge help, thank you! And, might I ask how you figured this out, since it was (afayk) undocumented?

No problem. API Monitor showed SetStdHandle(STD_OUTPUT_HANDLE, INVALID_HANDLE_VALUE) being called by the server which seemed odd to me. I looked at the server in hex rays and saw something along the lines of

Code: Select all

WCHAR buf[1024]; wsprintf(buf, L"\\\\.\\pipe\\%dcout", GetCurrentProcessId()); HANDLE hPipe = CreateFileW(buf,...); SetStdHandle(STD_OUTPUT_HANDLE, hPipe);

and recognised it as KFServer's attempt to use a named pipe handle for writing its output to if said pipe existed. (And if it didn't, it would pass the result to SetStdHandle anyway, which is where the INVALID_HANDLE_VALUE came from.)

The reason I think it's undocumented is because I can't find any relevant results for "pipe" "cout" "unreal engine", but that could just be my weak Google game.