RunCMD() v0.97 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

Post your working scripts, libraries and tools for AHK v1.1 and older
MrDoge
Posts: 161
Joined: 27 Apr 2020, 21:29

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

23 Oct 2022, 15:04

william_ahk wrote:
23 Oct 2022, 04:07
But yeah I do intend to read the stdout in real time too. My apologies for not describing it well. I'm trying to leverage the power of Python interactive console with AHK. Is there any other way? I probably have to resort to using ControlSend to cmd.exe

Code: Select all

#SingleInstance force
ListLines 0
KeyHistory 0
SendMode "Input" ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir A_ScriptDir ; Ensures a consistent starting directory.

f3::Exitapp

StrBuf(str, encoding)
{
    ; Calculate required size and allocate a buffer.
    buf := Buffer(StrPut(str, encoding))
    ; Copy or convert the string.
    StrPut(str, buf, encoding)
    return buf
}

class ChildProcess {
    static counter:=0

    __New(CmdLine, onStdOutCallback, includeStdErr:=false, WorkingDir:="") { ;from zig : std.ChildProcess.exec() : https://github.com/ziglang/zig/blob/4624c818991f161fc6a7021119e4d071b6e40e6c/lib/std/child_process.zig#L373
        ChildProcess.counter++
        pipe_path:="\\.\pipe\ahk-childprocess-" DllCall("GetCurrentProcessId") "-" ChildProcess.counter

        saAttr:=Buffer(24)
        NumPut("Uint",saAttr.Size,saAttr,0) ;nLength
        NumPut("Ptr",0,saAttr,8) ;lpSecurityDescriptor
        NumPut("Int",1,saAttr,16) ;bInheritHandle

        ;https://learn.microsoft.com/en-us/windows/win32/procthread/creating-a-child-process-with-redirected-input-and-output
        DllCall("CreatePipe"
            ,"Ptr*",&g_hChildStd_IN_Rd:=0
            ,"Ptr*",&g_hChildStd_IN_Wr:=0
            ,"Ptr",saAttr
            ,"Uint",0
        )

        bool:=DllCall("SetHandleInformation"
            ,"Ptr",g_hChildStd_IN_Wr
            ,"Uint",1 ;HANDLE_FLAG_INHERIT
            ,"Uint",0
        )

        g_hChildStd_OUT_Rd:=DllCall("CreateNamedPipe"
            ,"Str",pipe_path
            ,"Uint",0x40000001 ;0x00000001 | 0x40000000 ;PIPE_ACCESS_INBOUND | FILE_FLAG_OVERLAPPED
            ,"Uint",0 ;0x00000000 ;PIPE_TYPE_BYTE
            ,"Uint",1
            ,"Uint",4096
            ,"Uint",4096
            ,"Uint",0
            ,"Ptr",saAttr
        )

        g_hChildStd_OUT_Wr:=DllCall("CreateFile"
            ,"Str",pipe_path
            ,"Uint",0x40000000 ;GENERIC_WRITE
            ,"Uint",0
            ,"Ptr",saAttr
            ,"Uint",3 ;OPEN_EXISTING
            ,"Uint",0x80 ;FILE_ATTRIBUTE_NORMAL
            ,"Ptr",0
        )

        bool:=DllCall("SetHandleInformation"
            ,"Ptr",g_hChildStd_OUT_Rd
            ,"Uint",1 ;HANDLE_FLAG_INHERIT
            ,"Uint",0
        )

        P8 := (A_PtrSize==8)
        , SI:=Buffer(P8 ? 104 : 68, 0) ; STARTUPINFO structure
        , NumPut("UInt", P8 ? 104 : 68, SI) ; size of STARTUPINFO
        , NumPut("UInt", STARTF_USESTDHANDLES:=0x100, SI, P8 ? 60 : 44) ; dwFlags
        , NumPut("Ptr", g_hChildStd_IN_Rd, SI, P8 ? 80 : 56) ; hStdInput
        , NumPut("Ptr", g_hChildStd_OUT_Wr, SI, P8 ? 88 : 60) ; hStdOutput
        , (includeStdErr && NumPut("Ptr", g_hChildStd_OUT_Wr, SI, P8 ? 96 : 64)) ; hStdError
        , PI:=Buffer(P8 ? 24 : 16) ; PROCESS_INFORMATION structure

        bool:=DllCall("CreateProcess"
            ,"Ptr",0
            ,"Str",CmdLine
            ,"Ptr",0 ;lpProcessAttributes: *SECURITY_ATTRIBUTES
            ,"Int",0 ;lpThreadAttributes: *SECURITY_ATTRIBUTES
            ,"Int",True
            ,"Uint",0x08000400 ;dwCreationFlags: CREATE_NO_WINDOW=0x08000000, CREATE_UNICODE_ENVIRONMENT=0x00000400
            ,"Int",0 ;lpEnvironment
            ,"Ptr",WorkingDir ? StrPtr(WorkingDir) : 0
            ,"Ptr",SI
            ,"Ptr",PI
        )

        ; bool:=DllCall("CloseHandle", "Ptr",NumGet(PI,0,"Ptr")) ;piProcInfo.hProcess
        ; bool:=DllCall("CloseHandle", "Ptr",NumGet(PI,P8 ? 8 : 4,"Ptr")) ;piProcInfo.hThread

        bool:=DllCall("CloseHandle", "Ptr",g_hChildStd_OUT_Wr) ;THIS IS MUST, THIS IS IN FACT SO NECESSARY THAT WaitForSingleObject WILL ALWAYS HANG/BE WAITING
        bool:=DllCall("CloseHandle", "Ptr",g_hChildStd_IN_Rd)

        this.g_hChildStd_IN_Wr := g_hChildStd_IN_Wr
        this.g_hChildStd_OUT_Rd:=g_hChildStd_OUT_Rd
        this.onDataCallback := onDataCallback
        this.overlapped:=Buffer(32, 0)

        this.bump_amt:=512
        while (true) {
            this.next_buf:=Buffer(this.bump_amt)
            bool:=DllCall("ReadFile"
                ,"Ptr",g_hChildStd_OUT_Rd
                ,"Ptr",this.next_buf
                ,"Uint",this.next_buf.Size
                ,"Uint*",&read_bytes:=0
                ,"Ptr",this.overlapped
            )
            if (bool == 1) {
                this.bump_amt+=read_bytes
                this.onDataCallback.Call(StrGet(this.next_buf,read_bytes,"UTF-8"))
            } else {
                switch (A_LastError) {
                    case 997: ;IO_PENDING
                        break
                    case 109: ;BROKEN_PIPE: The pipe has been ended.
                        ; break outer1
                        return
                    default:
                        MsgBox "h89fh2398h4`nA_LastError: " A_LastError
                }
            }
        }

        this.checkForStr()
        this.timer := this.checkForStr.Bind(this)
        SetTimer this.timer, 10
    }

    checkForStr() {
        status:=DllCall("WaitForSingleObject"
                ,"Ptr",this.g_hChildStd_OUT_Rd
                ,"Uint",0
            )
        if (status == 258) {
            return
        } else if (status == 0) {
            bool:=DllCall("GetOverlappedResult"
                ,"Ptr",this.g_hChildStd_OUT_Rd
                ,"Ptr",this.overlapped
                ,"Uint*", &Overlapped_read_bytes:=0
                ,"Int",0
            )
            this.bump_amt+=Overlapped_read_bytes
            this.onDataCallback.Call(StrGet(this.next_buf,Overlapped_read_bytes,"UTF-8"))

            while (true) {
                this.next_buf:=Buffer(this.bump_amt)
                bool:=DllCall("ReadFile"
                    ,"Ptr",this.g_hChildStd_OUT_Rd
                    ,"Ptr",this.next_buf
                    ,"Uint",this.next_buf.Size
                    ,"Uint*",&read_bytes:=0
                    ,"Ptr",this.overlapped
                )
                if (bool == 1) {
                    this.bump_amt+=read_bytes
                    this.onDataCallback.Call(StrGet(this.next_buf,read_bytes,"UTF-8"))
                } else {
                    switch (A_LastError) {
                        case 997: ;IO_PENDING
                            break
                        case 109: ;BROKEN_PIPE: The pipe has been ended.
                            ; break outer1
                            SetTimer this.timer, 0
                            return
                        default:
                            MsgBox "h89fh2398h4`nA_LastError: " A_LastError
                    }
                }
            }
        } else {
            MsgBox "no way: " status
        }

    }

    noMoreStdIn() {
        bool:=DllCall("CloseHandle", "Ptr",this.g_hChildStd_IN_Wr)
    }

    writeToStdIn(str) {
        ; Calculate required size and allocate a buffer.
        wowBuf := Buffer(StrPut(str, "UTF-8"))
        ; Copy or convert the string.
        StrPut(str, wowBuf, "UTF-8")

        DllCall("WriteFile"
            ,"Ptr",this.g_hChildStd_IN_Wr
            ,"Ptr",wowBuf
            ,"Uint",wowBuf.Size - 1
            ,"Ptr*",&byteswritten:=0
            ,"Ptr",0
        )
    }
}

finalStr:=""
onDataCallback(str) {
    global finalStr

    finalStr.=str
    Tooltip finalStr
}
; -i is what took me so much time to figure out because no one had it: everyone was running python -u on SCRIPTS.py, not on STDIN
pythonProcess := ChildProcess("python -u -i", onDataCallback)
pythonProcess.writeToStdIn("print(2**20)`n")
Sleep 500
pythonProcess.writeToStdIn("print(2**19)`n")
pythonProcess.noMoreStdIn()
python is peculiar, I couldn't even make it work with node.js

Code: Select all

const child_process = require('child_process')

const pythonProcess = child_process.spawn('python')
pythonProcess.stdout.on('data', (data) => {
  process.stdout.write(data.toString())
})
pythonProcess.stderr.on('data', (data) => {
  process.stdout.write(data.toString())
})
pythonProcess.on('close', (code) => {
  process.stdout.write(`child process exited with code ${code}`)
})
function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms))
}
async function main() {
  pythonProcess.stdin.write("print(2**20)\n")
  await sleep(1000)
  pythonProcess.stdin.write("print(2**19)\n")
  // ONLY works when you end(), and it comes all at once, instead of in real time
  pythonProcess.stdin.end()
}
main()
but then I found out about -i in python --help

Code: Select all

const child_process = require('child_process')

const pythonProcess = child_process.spawn('python', ['-u', '-i'], {
  stdio: 'overlapped',
})
pythonProcess.stderr.pipe(pythonProcess.stdout)
pythonProcess.stdout.on('data', (data) => {
  process.stdout.write(data.toString())
})
if (true) { //you can comment this out to only get stdout
  pythonProcess.stderr.on('data', (data) => {
    process.stdout.write(data.toString())
  })
}
pythonProcess.on('close', (code) => {
  process.stdout.write(`child process exited with code ${code}`)
})
function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms))
}
async function main() {
  pythonProcess.stdin.write("print(2**20)\n")
  await sleep(1000)
  pythonProcess.stdin.write("print(2**19)\n")
  // doesn't need end()
}
main()
william_ahk
Posts: 496
Joined: 03 Dec 2018, 20:02

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

23 Oct 2022, 23:55

@TheArkive So the ControlSend technique is already fully implemented, very nice!
@MrDoge Thank you so much! :thumbup: You could release it as a library. I think it's a well important component that has many use cases.
AldoLodos
Posts: 2
Joined: 09 Apr 2022, 06:56

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

26 Dec 2022, 11:34

Hello,

I'm using RUNCMD in a script and it works generally fine, but sometimes it runs in a infinite loop :

264: While,A_Args.RunCMD.PID && (Line := File.ReadLine())
262: While,(A_Args.RunCMD.PID + DllCall("Sleep", "Int",0)) && DllCall("PeekNamedPipe", "Ptr",hPipeR, "Ptr",0, "Int",0, "Ptr",0, "Ptr",0, "Ptr",0)
264: While,A_Args.RunCMD.PID && (Line := File.ReadLine())
262: While,(A_Args.RunCMD.PID + DllCall("Sleep", "Int",0)) && DllCall("PeekNamedPipe", "Ptr",hPipeR, "Ptr",0, "Int",0, "Ptr",0, "Ptr",0, "Ptr",0)
264: While,A_Args.RunCMD.PID && (Line := File.ReadLine())
262: While,(A_Args.RunCMD.PID + DllCall("Sleep", "Int",0)) && DllCall("PeekNamedPipe", "Ptr",hPipeR, "Ptr",0, "Int",0, "Ptr",0, "Ptr",0, "Ptr",0)
....

Any idea on what could happen ?
Thanks
juno77
Posts: 2
Joined: 14 Jan 2023, 16:54

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

14 Jan 2023, 17:02

Hello,
with the stable version 2.0.2 I thought about migrating to v2. But there is an error in compilung RunCMD(). Is there already a migrated version for RunCMD() available?
Thanks
neogna2
Posts: 591
Joined: 15 Sep 2016, 15:44

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

15 Jan 2023, 08:14

juno77 wrote:
14 Jan 2023, 17:02
with the stable version 2.0.2 I thought about migrating to v2. But there is an error in compilung RunCMD(). Is there already a migrated version for RunCMD() available?
SKAN was waiting for an issue but released a temp v2 version on page 10 of this thread. There is also a v2 version called RunTerminal made by another person. Both have the issue, see discussions in posts starting here and forward to see if that issue affects your use case or not.
juno77
Posts: 2
Joined: 14 Jan 2023, 16:54

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

15 Jan 2023, 15:38

Thanks a lot for the hints. I missed this temp v2 version :crazy: .
Compilation is no problem any more. But runCMD returns to early because B_OK is 0 (line 62 of temp RunCMD).
Propably I have to dig a bit deaper to this problem, or do you have an idea where to search?
iruizdeconejo
Posts: 9
Joined: 24 Jan 2023, 19:35

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

07 Mar 2023, 18:50

Hi,
The temp version for V2 is working for me, except that I´m not getting any lines.
When debugging the program, I see that the code runs up till
While Glob.RunCMD.PID
then jumps to line
While Glob.RunCMD.PID and not FileObj.AtEOF
and continues in the first While statement.
To me, the obvious thing is that FileObj.AtEOF is True and therefore the callback is not being executed, despite the fact that the program continues and the expected result is ok.
How can that be?

The command line I am issuing, when passed to StdoutToVar instead of RunCMd, takes 40 seconds and returns a 30 lines text.

Thanks in advance,
Ignacio
MrDoge
Posts: 161
Joined: 27 Apr 2020, 21:29

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

07 Mar 2023, 21:03

I don't know if I've posted this before, try #1 : if there's the newline problem, #2 : if there's the B_OK problem
, there's a catch to this version : once ahk exits, the child_process is killed, this behavior may be desired, or not.

Code: Select all

#SingleInstance force
ListLines 0
KeyHistory 0
SendMode "Input" ; Recommended for new scripts due to its superior speed and reliability.
SetWorkingDir A_ScriptDir ; Ensures a consistent starting directory.

MsgBox child_process_killChildOnExit("Where user32.dll")

child_process_killChildOnExit(CmdLine, WorkingDir:="") { ;from zig : std.ChildProcess.exec() : https://github.com/ziglang/zig/blob/4624c818991f161fc6a7021119e4d071b6e40e6c/lib/std/child_process.zig#L373
    if (!child_process_killChildOnExit.HasOwnProp("counter")) {
        ; https://stackoverflow.com/questions/3342941/kill-child-process-when-parent-process-is-killed
        child_process_killChildOnExit.counter := 0
        child_process_killChildOnExit.s_jobHandle:=DllCall("CreateJobObject"
            ,"Ptr",0
            ,"Str","ChildProcessTracker" DllCall("GetCurrentProcessId")
        )
        ; doesn't work somehow..
        ; info := Buffer(64, 0) ;JOBOBJECT_BASIC_LIMIT_INFORMATION
        ; NumPut("Uint",0x2000,info,16) ; LimitFlags: 0x2000: JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
        ; bool:=DllCall("SetInformationJobObject"
            ; ,"Ptr",child_process_killChildOnExit.s_jobHandle
            ; ,"int",2 ;JOBOBJECT_BASIC_LIMIT_INFORMATION
            ; ,"Ptr",info
            ; ,"Uint", 64 ;sizeof_JOBOBJECT_BASIC_LIMIT_INFORMATION
        ; )
        extendedInfo := Buffer(144, 0) ;JOBOBJECT_EXTENDED_LIMIT_INFORMATION
        NumPut("Uint",0x2000,extendedInfo,16) ; LimitFlags: 0x2000: JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
        bool:=DllCall("SetInformationJobObject"
            ,"Ptr",child_process_killChildOnExit.s_jobHandle
            ,"int",9 ;JOBOBJECT_EXTENDED_LIMIT_INFORMATION
            ,"Ptr",extendedInfo
            ,"Uint", 144 ;sizeof_JOBOBJECT_EXTENDED_LIMIT_INFORMATION
        )
    }

    child_process_killChildOnExit.counter++
    pipe_path:="\\.\pipe\ahk-childprocess-" DllCall("GetCurrentProcessId") "-" child_process_killChildOnExit.counter

    saAttr:=Buffer(24)
    NumPut("Uint",saAttr.Size,saAttr,0) ;nLength
    NumPut("Ptr",0,saAttr,8) ;lpSecurityDescriptor
    NumPut("Int",1,saAttr,16) ;bInheritHandle

    read_handle:=DllCall("CreateNamedPipe"
        ,"Str",pipe_path
        ,"Uint",0x40000001 ;0x00000001 | 0x40000000 ;PIPE_ACCESS_INBOUND | FILE_FLAG_OVERLAPPED
        ,"Uint",0 ;0x00000000 ;PIPE_TYPE_BYTE
        ,"Uint",1
        ,"Uint",4096
        ,"Uint",4096
        ,"Uint",0
        ,"Ptr",saAttr
    )

    write_handle:=DllCall("CreateFile"
        ,"Str",pipe_path
        ,"Uint",0x40000000 ;GENERIC_WRITE
        ,"Uint",0
        ,"Ptr",saAttr
        ,"Uint",3 ;OPEN_EXISTING
        ,"Uint",0x80 ;FILE_ATTRIBUTE_NORMAL
        ,"Ptr",0
    )

    bool:=DllCall("SetHandleInformation"
        ,"Ptr",read_handle
        ,"Uint",1 ;HANDLE_FLAG_INHERIT
        ,"Uint",0
    )

    P8 := (A_PtrSize==8)
    , SI:=Buffer(P8 ? 104 : 68, 0) ; STARTUPINFO structure
    , NumPut("UInt", P8 ? 104 : 68, SI) ; size of STARTUPINFO
    , NumPut("UInt", STARTF_USESTDHANDLES:=0x100, SI, P8 ? 60 : 44) ; dwFlags
    , NumPut("Ptr", write_handle, SI, P8 ? 88 : 60) ; hStdOutput
    , NumPut("Ptr", write_handle, SI, P8 ? 96 : 64) ; hStdError
    , PI:=Buffer(P8 ? 24 : 16) ; PROCESS_INFORMATION structure

    bool:=DllCall("CreateProcess"
        ,"Ptr",0
        ,"Str",CmdLine
        ,"Ptr",0 ;lpProcessAttributes: *SECURITY_ATTRIBUTES
        ,"Int",0 ;lpThreadAttributes: *SECURITY_ATTRIBUTES
        ,"Int",True
        ,"Uint",0x08000400 ;dwCreationFlags: CREATE_NO_WINDOW=0x08000000, CREATE_UNICODE_ENVIRONMENT=0x00000400
        ,"Int",0 ;lpEnvironment
        ,"Ptr",WorkingDir ? StrPtr(WorkingDir) : 0
        ,"Ptr",SI
        ,"Ptr",PI
    )

    ;<ADD IT
    hProcess:=NumGet(PI,0,"Ptr")

    DllCall("AssignProcessToJobObject"
        ,"Ptr",child_process_killChildOnExit.s_jobHandle
        ,"Ptr",hProcess
    )
    ;ADD IT>

    bool:=DllCall("CloseHandle", "Ptr",write_handle) ;THIS IS MUST, THIS IS IN FACT SO NECESSARY THAT WaitForSingleObject WILL ALWAYS HANG/BE WAITING

    overlapped:=Buffer(32, 0)

    bump_amt:=512
    finalStr:=""

    outer1:
    while (true) {
        while (true) {
            next_buf:=Buffer(bump_amt)
            bool:=DllCall("ReadFile"
                ,"Ptr",read_handle
                ,"Ptr",next_buf
                ,"Uint",next_buf.Size
                ,"Uint*",&read_bytes:=0
                ,"Ptr",overlapped
            )
            if (bool == 1) {
                bump_amt+=read_bytes
                finalStr.=StrGet(next_buf,read_bytes,"UTF-8")
            } else {
                switch (A_LastError) {
                    case 997: ;IO_PENDING
                        break
                    case 109: ;BROKEN_PIPE: The pipe has been ended.
                        break outer1
                    default:
                        MsgBox "h89fh2398h4`nA_LastError: " A_LastError
                }
            }
        }
        ; status:=DllCall("WaitForSingleObject"
        ;     ,"Ptr",read_handle
        ;     ,"Uint",4294967295 ;INFINITE ;If dwMilliseconds is INFINITE, the function will return only when the object is signaled.
        ; )
        while (true) { ;this is faster but maybe more expensive???
            status:=DllCall("WaitForSingleObject"
                ,"Ptr",read_handle
                ,"Uint",0
            )
            if (status == 258) {
                DllCall("Sleep", "Uint",0)
                continue
            } else if (status == 0) {
                break
            } else {
                MsgBox "no way: " status
            }
        }

        bool:=DllCall("GetOverlappedResult"
            ,"Ptr",read_handle
            ,"Ptr",overlapped
            ,"Uint*", &Overlapped_read_bytes:=0
            ,"Int",0
        )
        bump_amt+=Overlapped_read_bytes
        finalStr.=StrGet(next_buf,Overlapped_read_bytes,"UTF-8")

    }

    return finalStr
}


return

f3::Exitapp
Adventurer
Posts: 23
Joined: 18 Apr 2019, 06:24

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

10 Jul 2023, 13:23

SKAN wrote:
01 Mar 2022, 17:38
@Finallf
@neogna2

Here is a temp version for V2

...
Given AHK2 is now stable, you should post this up as an official, non-temp release in the v2 Scripts forum.

The only other non-blocking stdout capturing function I could find for AHK2 is child_process, and it's rather advanced.
User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

11 Jul 2023, 04:15

teadrinker wrote:
06 Apr 2022, 02:15
SKAN wrote:

Code: Select all

While A_Args.RunCMD.PID and (Line := File.ReadLine())
What if Line is "0" ?
:(
 
Corrected it to

Code: Select all

        While A_Args.RunCMD.PID and StrLen(Line := File.ReadLine())
Thanks. :)
User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: RunCMD() v0.94 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

11 Jul 2023, 04:18

Adventurer wrote:
10 Jul 2023, 13:23
SKAN wrote:
01 Mar 2022, 17:38
@Finallf
@neogna2

Here is a temp version for V2

...
Given AHK2 is now stable, you should post this up as an official, non-temp release in the v2 Scripts forum.
Yes. ASAP.
Ref: [V2] FileObj.ReadLine() autotrims trailing LF/CRLF
teadrinker
Posts: 4336
Joined: 29 Mar 2015, 09:41
Contact:

Re: RunCMD() v0.96 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

13 Jul 2023, 15:47

I noticed one more drawback. Your function loads the processor by 10 - 12%.
User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: RunCMD() v0.96 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

14 Jul 2023, 00:44

teadrinker wrote:
13 Jul 2023, 15:47
Your function loads the processor by 10 - 12%.
 
Its owing to high nested loop iteration.
With SetbatchLines -1 in effect, ping www.autohotkey.com -n 4 takes over 400000 iterations to complete.
By changing the sleep value from 0 to 1 in While ( A_Args.RunCMD.PID | DllCall("Sleep", "Int",0) ) it takes a mere 200+ iterations to complete,
though minimally slower.

I am already handling this in temp version for V2 with an additional parameter P_Slow as no SetBatchlines in V2.
I guess I'll implement that parameter for V1.1 version too.

Edit: RunCMD() updated to v0.97
teadrinker
Posts: 4336
Joined: 29 Mar 2015, 09:41
Contact:

Re: RunCMD() v0.96 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

14 Jul 2023, 07:04

I've tried to write code (v2) for asynchronous reading of stdout using a different approach. It turned out to be a bit cumbersome, but it lacks the above-mentioned disadvantage and is really non-blocking: :)

Code: Select all

#Requires AutoHotkey v2

inst := ''
F3::    ; F3 without timeout
F4:: {  ; F4 timeout 2 sec
    ReadOutput('', '')
    try global inst := AsyncStdoutReader('ping -n 8 google.com', ReadOutput, A_ThisHotkey = 'F3' ? unset : 2000)
    catch MethodError as e {
        MsgBox A_Clipboard := e.Stack
        MsgBox e.Message . '`n' . e.What . '`nLine: ' e.Line
        ExitApp
    }
    MsgBox 'AsyncStdoutReader is non-blocking', 'test', 0x1000 . ' T2'
}

ReadOutput(line, complete := 0) {
    global inst
    static EM_SETSEL := 0xB1, myGui := '', text := '', edit := ''
    if !myGui {
        myGui := Gui('+Resize', 'Async reading of stdout')
        myGui.MarginX := myGui.MarginY := 0
        myGui.SetFont('s12', 'Consolas')
        myGui.AddText('x10 y10', 'Complete: ')
        text := myGui.AddText('x+5 yp w100', 'false')
        edit := myGui.AddEdit('xm y+10 w650 h500')
        edit.GetPos(, &y := unset)
        myGui.OnEvent('Size', (o, m, w, h) => edit.Move(,, w, h - y))
        myGui.OnEvent('Close', (*) => ExitApp())
        myGui.Show()
    }
    (line = '' && complete = '' && edit.Value := '')
    text.Value := complete = -1 ? 'timed out' : complete = false ? 'false' : 'true'
    SendMessage EM_SETSEL, -2, -1, edit
    EditPaste line, edit

    if inst && complete {
        outData := inst.outData, inst := ''
        MsgBox outData, 'Complete stdout', 0x2040
    }
}

class AsyncStdoutReader
{
    __New(cmd, callback?, timeout?, encoding?) {
        encoding := encoding ?? 'cp' . DllCall('GetOEMCP', 'UInt')
        this.event := %this.__Class%.Event()
        this.params := {
            buf: Buffer(4096, 0),
            overlapped: Buffer(A_PtrSize * 3 + 8, 0),
            hEvent: this.event.handle,
            outData: '',
            encoding: encoding,
            complete: false
        }
        (IsSet(callback) && this.params.callback := callback)
        if IsSet(timeout) {
            this.params.timeout := timeout
            this.params.startTime := A_TickCount
        }
        this.process := %this.__Class%.Process(cmd, this.params)
        this.signal := %this.__Class%.EventSignal(this.process, this.params)
        this.process.Read()
    }

    processID => this.process.PID

    complete => this.params.complete

    outData => this.params.outData

    __Delete() {
        DllCall('CancelIoEx', 'Ptr', this.process.hPipeRead, 'Ptr', this.params.overlapped)
        this.event.Set()
        this.signal.Clear()
        this.process.Clear()
        this.params.buf.Size := 0
        this.params.outData := ''
    }

    class Event
    {
        __New() => this.handle := DllCall('CreateEvent', 'Int', 0, 'Int', 0, 'Int', 0, 'Int', 0, 'Ptr')
        __Delete() => DllCall('CloseHandle', 'Ptr', this.handle)
        Set() => DllCall('SetEvent', 'Ptr', this.handle)
    }

    class Process
    {
        __New(cmd, info) {
            this.info := info
            this.CreatePipes()
            if !this.PID := this.CreateProcess(cmd) {
                throw OSError('Failed to create process')
            }
        }

        CreatePipes() {
            static FILE_FLAG_OVERLAPPED := 0x40000000, PIPE_ACCESS_INBOUND := 0x1
                 , pipeMode := (PIPE_TYPE_BYTE := 0) | (PIPE_WAIT := 0)
                 , GENERIC_WRITE := 0x40000000, OPEN_EXISTING := 0x3
                 , FILE_ATTRIBUTE_NORMAL := 0x80, HANDLE_FLAG_INHERIT := 0x1

            this.hPipeRead := DllCall('CreateNamedPipe', 'Str', pipeName := '\\.\pipe\StdOut_' . A_TickCount,
                'UInt', PIPE_ACCESS_INBOUND | FILE_FLAG_OVERLAPPED, 'UInt', pipeMode, 'UInt', 1,
                'UInt', this.info.buf.Size, 'UInt', this.info.buf.Size, 'UInt', 120000, 'Ptr', 0, 'Ptr')

            this.hPipeWrite := DllCall('CreateFile', 'Str', pipeName, 'UInt', GENERIC_WRITE, 'UInt', 0, 'Ptr', 0,
                'UInt', OPEN_EXISTING, 'UInt', FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED, 'Ptr', 0, 'Ptr')
            DllCall('SetHandleInformation', 'Ptr', this.hPipeWrite, 'UInt', HANDLE_FLAG_INHERIT, 'UInt', HANDLE_FLAG_INHERIT)
        }

        CreateProcess(cmd) {
            static STARTF_USESTDHANDLES := 0x100, CREATE_NO_WINDOW := 0x8000000
            STARTUPINFO := Buffer(siSize := A_PtrSize * 9 + 4 * 8, 0)
            NumPut('UInt', siSize, STARTUPINFO)
            NumPut('UInt', STARTF_USESTDHANDLES, STARTUPINFO, A_PtrSize * 4 + 4 * 7)
            NumPut('Ptr', this.hPipeWrite, 'Ptr', this.hPipeWrite, STARTUPINFO, siSize - A_PtrSize * 2)

            PROCESS_INFORMATION := Buffer(A_PtrSize * 2 + 4 * 2, 0)
            if !DllCall('CreateProcess', 'Ptr', 0, 'Str', cmd, 'Ptr', 0, 'Ptr', 0, 'UInt', true,
                'UInt', CREATE_NO_WINDOW, 'Ptr', 0, 'Ptr', 0, 'Ptr', STARTUPINFO, 'Ptr', PROCESS_INFORMATION)
                Return this.Clear()
            DllCall('CloseHandle', 'Ptr', this.hPipeWrite), this.hPipeWrite := 0
            Return PID := NumGet(PROCESS_INFORMATION, A_PtrSize * 2, 'UInt')
        }

        Read() {
            buf := this.info.buf, overlapped := this.info.overlapped
            overlapped.__New(overlapped.Size, 0)
            NumPut('Ptr', this.info.hEvent, overlapped, A_PtrSize * 2 + 8)
            bool := DllCall('ReadFile', 'Ptr', this.hPipeRead, 'Ptr', buf, 'UInt', buf.Size, 'UIntP', &size := 0, 'Ptr', overlapped)
            if bool {
                this.info.outData .= str := StrGet(buf, size, this.info.encoding)
                (this.info.HasProp('callback') && SetTimer(this.info.callback.Bind(str), -10))
                this.Read()
            }
            else if !bool && A_LastError != ERROR_IO_PENDING := 997 {
                this.info.complete := true
                (this.info.HasProp('callback') && SetTimer(this.info.callback.Bind('', true), -10))
            }
        }

        Clear() {
            DllCall('CloseHandle', 'Ptr', this.hPipeRead)
            (this.hPipeWrite && DllCall('CloseHandle', 'Ptr', this.hPipeWrite))
        }
    }

    class EventSignal
    {
        __New(stdOut, info) {
            this.info := info
            this.stdOut := stdOut
            this.onEvent := ObjBindMethod(this, 'Signal')
            timeout := info.HasProp('timeout') ? info.timeout : -1
            this.regWait := this.RegisterWaitCallback(this.info.hEvent, this.onEvent, timeout)
        }

        Signal(handle, timedOut) {
            if timedOut {
                (this.info.HasProp('callback') && SetTimer(this.info.callback.Bind('', -1), -10))
                return
            }
            if !DllCall('GetOverlappedResult', 'Ptr', handle, 'Ptr', this.info.overlapped, 'UIntP', &size := 0, 'UInt', false) {
                (this.info.HasProp('callback') && SetTimer(this.info.callback.Bind('', true), -10))
                return this.info.complete := true
            }
            this.info.outData .= str := StrGet(this.info.buf, size, this.info.encoding)
            (this.info.HasProp('callback') && SetTimer(this.info.callback.Bind(str), -10))
            this.stdOut.Read()
            timeout := this.info.HasProp('timeout') ? this.info.timeout - A_TickCount + this.info.startTime : -1
            this.regWait := this.RegisterWaitCallback(this.info.hEvent, this.onEvent, timeout)
        }

        Clear() {
            this.regWait.Unregister()
            this.DeleteProp('regWait')
            this.DeleteProp('onEvent')
        }

        RegisterWaitCallback(handle, callback, timeout := -1) {
            ; by lexikos https://www.autohotkey.com/boards/viewtopic.php?t=110691
            static waitCallback, postMessageW, wnd, nmsg := 0x5743
            if !IsSet(waitCallback) {
                if A_PtrSize = 8 {
                    NumPut('int64', 0x8BCAB60F44C18B48, 'int64', 0x498B48C18B4C1051, 'int64', 0x20FF4808, waitCallback := Buffer(24))
                } else {
                    NumPut('int64', 0x448B50082444B60F, 'int64', 0x70FF0870FF500824, 'int64', 0x0008C2D0FF008B04, waitCallback := Buffer(24))
                }
                DllCall('VirtualProtect', 'ptr', waitCallback, 'ptr', 24, 'uint', 0x40, 'uint*', 0)
                postMessageW := DllCall('GetProcAddress', 'ptr', DllCall('GetModuleHandle', 'str', 'user32', 'ptr'), 'astr', 'PostMessageW', 'ptr')
                wnd := Gui(), DllCall('SetParent', 'ptr', wnd.hwnd, 'ptr', -3)    ; HWND_MESSAGE = -3
                OnMessage(nmsg, messaged, 255)
            }
            NumPut('ptr', postMessageW, 'ptr', wnd.hwnd, 'uptr', nmsg, param := %StrSplit(this.__Class, '.')[1]%.EventSignal.RegisteredWait())
            NumPut('ptr', ObjPtr(param), param, A_PtrSize * 3)
            param.callback := callback, param.handle := handle
            if !DllCall('RegisterWaitForSingleObject', 'ptr*', &waitHandle := 0,
                'ptr', handle, 'ptr', waitCallback, 'ptr', param, 'uint', timeout, 'uint', 8)
                throw OSError()
            param.waitHandle := waitHandle, param.locked := ObjPtrAddRef(param)
            return param
            static messaged(wParam, lParam, nmsg, hwnd) {
                if hwnd = wnd.hwnd {
                    local param := ObjFromPtrAddRef(NumGet(wParam + A_PtrSize * 3, 'ptr'))
                    (param.callback)(param.handle, lParam)
                    param._unlock()
                }
            }
        }

        class RegisteredWait extends Buffer
        {
            static prototype.waitHandle := 0, prototype.locked := 0
            __new() => super.__new(A_PtrSize * 5, 0)
            __delete() => this.Unregister()
            _unlock() {
                (p := this.locked) && (this.locked := 0, ObjRelease(p))
            }
            Unregister() {
                wh := this.waitHandle, this.waitHandle := 0
                (wh) && DllCall('UnregisterWaitEx', 'ptr', wh, 'ptr', -1)
                this._unlock()
            }
        }
    }
}
Last edited by teadrinker on 14 Jul 2023, 17:03, edited 2 times in total.
User avatar
SKAN
Posts: 1551
Joined: 29 Sep 2013, 16:58

Re: RunCMD() v0.96 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

14 Jul 2023, 10:02

teadrinker wrote:
14 Jul 2023, 07:04
and is really non-blocking: :)
Nice! :thumbup: :)

Nitpick:

Code: Select all

        RegisterWaitCallback(handle, callback, timeout := -1) {
            ; by lexicos https://www.autohotkey.com/boards/viewtopic.php?t=110691
RegisterWaitCallback() is by lexikos (not lexicos)
Starmina
Posts: 1
Joined: 15 May 2020, 05:51

Re: RunCMD() v0.97 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

28 Sep 2023, 16:47

Hi, thanks for this very nice script !

Does anyone have the issue where there's random line return in the output string ?
I have a loop that return a simple string, and it'll sometimes add random line return in the string ??
I tried another script that simply output an ADB (android debug bridge) command, and the issue appear from time to time too, the output string will contain a line return without any reason.
I can't reproduce it. It just does it from time to time.
Marium0505
Posts: 40
Joined: 11 May 2020, 20:45

Re: RunCMD() v0.97 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

02 Oct 2023, 06:17

@teadrinker
This works great, thanks a lot, and it deserves a thread in the Scripts and Functions (v2) forum!

A question about the timeout, is the timeout dependent on time since return of the previous line or since the function was run? Seems like the first one due to these two lines in Signa()?

Code: Select all

			timeout := this.info.HasProp('timeout') ? this.info.timeout - A_TickCount + this.info.startTime : -1
			this.regWait := this.RegisterWaitCallback(this.info.hEvent, this.onEvent, timeout)
I would love to be interested in something similar for Powershell.

Thanks again, and nice work. :clap:
teadrinker
Posts: 4336
Joined: 29 Mar 2015, 09:41
Contact:

Re: RunCMD() v0.97 : Capture stdout to variable. Non-blocking version. Pre-process/omit individual lines.

02 Oct 2023, 15:30

Marium0505 wrote: and it deserves a thread in the Scripts and Functions (v2) forum!
Maybe when I have time! :)
The timeout in this case is simply the maximum time the class object will work with the command line. When it expires, waiting for new Stdout data stops.

Return to “Scripts and Functions (v1)”

Who is online

Users browsing this forum: gwarble, xavierarmand and 113 guests