AutoHotkey Community

The beauty of the Windows (or Unix) batch language is that it is almost infinitely extensible via external utilities that take input from their command line or from the standard input device and send results to the standard output device. It is not merely its ability to conveniently launch such executables that is the strength of the batch language, it is also its ability to efficiently capture their ouput for further processing (e.g. the FOR /F command, piping, etc.)

So what about AutoHotkey? Even if some other extension mechanism for AutoHotkey were to become available (invoking DLLs, making API or ActiveX calls, etc.), it seems likely that external console utilities will always be a big part of our scripting arsenal, and I for one would love to see their use in AutoHotkey become more convenient and more efficient. Currently, the typical scenario (in which the contents of one variable needs to be passed as standard input to an external utility and another variable needs to eventually contain the utility's standard output) requires the use of disk files and of the command processor, something along the lines of

FileAppend, %InVar%, InFile.tmp
RunWait, %ComSpec% /C Utility.exe < InFile.tmp > OutFile.tmp,, Hide
FileRead, OutVar, OutFile.tmp

I know, this is the most general case -- in the special case where the input consists of a single text line, an input file may be avoided and we can get away with

RunWait, %ComSpec% /C Echo %InVar%| Utility.exe > OutFile.tmp,, Hide
FileRead, OutVar, OutFile.tmp

Either way, we still need to use disk files and the command processor (and I'm not even mentioning the extra lines required to manage and clean up the temporary files used). Even such a valient alternative as corrupt's CmdRet utility (decribed elsewhere in these forums) requires extra machinery (creating a GUI with an Edit control, etc).

So, what I'm suggesting (if it hasn't been suggested before) is an extension of RunWait to allow the direct specification of input and/or output variables corresponding to the target executable's STDIN and STDOUT. The actual mechanism used to implement this need not be anything too fancy, maybe just copying the variables' contents to and from in-memory temporary files and providing those filespecs as the external program's STDIN and STDOUT. The point is to integrate external "helper" utilities into AutoHotkey in as natural a fashion as possible. [If you feel ambitious, providing a redirection capability for STDERR, like CMD's 2> feature, wouldn't be a bad thing either! ]

Anyway, that's my pitch.

Thanks,

Jacques.

Mr. Speaker, I ask for unanimous consent to revise and extend my remarks.

It occurs to me that any feature suggestion should in fairness be accompanied by a corresponding syntax suggestion so all can judge whether the feature can be implemented elegantly and without affecting existing code, so one possibility that comes to mind in this case is to extend the "options" parameter of RunWait (where Max, Min, Hide and UseErrorLevel are currently specified), maybe something like this:

RunWait, Utility.exe,, Hide in:InVar out:OutVar err:ErrVar

The new syntax elements would of course be optional.

Thanks again,

Jacques.

Would be great indeed, see this posting...

_________________
mirror 1 • mirror 2 • mirror 3 • ahk4.me • PM or

One more (last?) comment:

As with obtaining a resulting process ID, the ability to specify an executable's standard input, output and error handles will be limited to that majority of cases where the RunWait implementation is able to use CreateProcess() rather than ShellExecute(). This is perfectly acceptable! Not trying to make your life difficult , just trying to bring all those great console utilities available out there (and ones yet to be written as the need arises) into AutoHotkey as practically native commands. This could even make certain AutoHotkey feature requests unnecessary or less urgent (when the right external utility already exists or can be written), relieving you to work on the hard stuff!

I'd offer to help, but I never went beyond Assembler or plain C [i.e. life's too short for C++ and OOP ] This means I can only extend AutoHotkey externally by writing custom utilities. The AutoHotkey source code is just a fascinating mystery to me!

Thanks,

Jacques.

Ah, hadn't noticed that posting. Great minds think alike, I guess.

You of course went beyond the redirection request for RunWait and added the "live interprocess piping" idea. Probably scared Chris off!

Jacques.

That looks like a good approach. I guess they could also be new parameters at the end.

Good point.

Since you're more knowledgeable in this area, you've probably already verified that this such a feature is definitely possible using standard API calls. If so, having some plain C source code, or even just a list of how and when to call the API functions, would help increase my confidence and thus make it much easier to implement this sooner.

Corrupt was nice enough to provide the source code for CMDret, which is high on the list to try to integrate into AutoHotkey. However, I'm not sure of exactly how much it overlaps with your proposal.

Totally your call! I wasn't sure if you had or were planning some additional "future use" parameters on the trailing end of RunWait.


Yes. Redirection of STDIN, STDOUT and STDERR is supported directly by CreateProcess() via the last three members of its STARTUPINFO structure argument (HANDLE hStdInput; HANDLE hStdOutput; HANDLE hStdError;). The dwFlags member of the structure must also have the flag STARTF_USESTDHANDLES set in order for the three redirection handles to be used -- they are ignored otherwise -- so you can add this flag only if some redirection is being requested. If I'm reading your code correctly (script.cpp), your STARTUPINFO structure for calls to CreateProcess() is named "si" -- you'd just need to set the extra entries to make this work (you can use the GetStdHandle() function to obtain the current in/out/err handles in order to correctly set those among the three handles for which redirection is not being requested).

This just leaves the question of what you are redirecting to: from the AutoHotkey programmer's standpoint, redirection is to and/or from AutoHotkey variables, but the API of course requires handles to some sort of file system objects. You may have ideas about this, but off-hand it seems to me that using randomly named temporary files (see the GetTempFileName() function to generate random names and GetTempPath() to get the path to use) created using the FILE_ATTRIBUTE_TEMPORARY flag may combine simplicity and efficiency (because files flagged as TEMPORARY usually remain in memory and may never actually get written to disk if they are deleted after use). The idea would be to copy the contents of the in variable to such a temp file before invoking CreateProcess() and to copy the contents of the output and/or error temp files back to the specified out and/or err variables after the process terminates. If anybody has other ideas about this (e.g. in-memory files?), please speak up.

I figure that this will mostly make sense only for RunWait, though redirecting input for Run commands might also be useful -- I'm just not sure how you could get back the output of executables launched via Run rather than RunWait -- maybe on a timed basis as daonlyfreez suggested in a previous thread?

Anyway, I'm rambling a little -- Please feel free to ask for any clarification you may need.

Thanks,

Jacques.

Looking at corrupt's CMDret source code, I see that he uses a handle to an anonymous pipe rather than to a file as the target for the launched program's redirected output. This looks like a very neat approach (you'd need three separate pipes to provide for redirection of STDIN, STDOUT and STDERR), but would require more work on your part to accumulate all of the launched program's output because, unlike a file, a pipe's size does not grow arbitrarily -- A pipe is created with a fixed buffer size, and write operations on one end of the pipe will remain in a blocked state (i.e. not return) until all of the data is written, which will require read operations on the other end of the pipe to free up room in the buffer in cases where the amount of data to be written is larger than the selected buffer size. The problem is that, while it will be easy to pick a buffer size for the pipe used to provide input to the lanched program large enough to provide it all in one shot (since the size of the input variable will be known to AutoHotkey), picking a buffer size for the pipes that will receive output and error output from the executable large enough to allow AutoHotkey to get the output in a single read would require advance knowledge of the maximum size of the executable's output (which may not be known even to the programmer).

In any case, if you wish to explore pipes rather than files to achieve redirection, here's a link to a short and sweet MSDN discussion, complete with sample source code: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/creating_a_child_process_with_redirected_input_and_output.asp

By the way, regarding your question as to the degree to which my feature request overlaps the CMDret idea, I think they have two separate purposes: my suggestion relates to the need for a simple mechanism to integrate external utilities into AutoHotkey programs by providing a "single-line" way to both launch a utility and "map" the utility's input and output to AutoHotkey variables. CMDret, on the other hand, provides a mechanism that allows the gradual display (or accumulation) of a launched program's standard output within a GUI edit control, a way of "taming" the output of an external program by capturing it (and optionally displaying it) within a GUI control belonging to the AutoHotkey program. Of course, Cmdret could be used to quietly capture a program's output (by keeping the GUI or its edit control hidden), but it's too complicated a mechanism for this purpose (it would be easier and require less setup work to continue explicitly using files and the command processor to achieve redirection, as described in my first posting in this thread).

Later,

Jacques.

Although CMDret uses SendMessage with EM_REPLACESEL to update a specified control's contents it could just as easily store the data to an internal variable instead without using a control... The main reason for using a control was to send the data back to AHK but if the variable's value is already in AHK...

If an eventual "built-in" implementation of CMDret within AutoHotkey uses variables rather than controls, then it becomes essentially the same feature as we've been discussing in this thread, do you agree? What's your sense of what CMDret would look like if built into AHK? Maybe we're asking for the same thing.

Jacques.

The documentation is cloudy about this: Even if the file stays in memory, it might still get mirrored to disk, and thus require real disk space and cause real wear on the disk (which might be important for huge files or for operations that are run thousands of times per day).

Thus, I will probably look into the pipes idea more in the hope that it can be adapted without greatly increasing the code complexity.

Thanks for all the details. They should definitely give me a head start when the time comes to work on this (hopefully within the next month). If anyone has more info or can do more research about the feasiblity of redirecting input and output entirely in memory (such as with pipes), please post it here or e-mail it to support@autohotkey.com

The discussion of console I/O redirection via pipes in the following MSKB article seems a little more complete than the one in the MSDN article I mentioned in a previous message in this thread:

http://support.microsoft.com/default.aspx?scid=kb;en-us;190351

One thing about the sample program in this article: rather than provide for separate pipes for the child program's normal output and error output, it provides a duplicate of the output pipe handle for use also as the error output handle. Providing separate pipes for the two is a simple modification.

If pipes seem overly complex, I continue to believe that a simpler implementation based on temporary files flagged with FILE_ATTRIBUTE_TEMPORARY (also described somewhere in this thread) would be quite satisfactory. Even if such files were to occasionally actually hit the disk, the performance would at worst still be better than the only currently available mechanism (which, as described in the leadoff post in this thread, is also file based, requires much more setup and cleanup machinery, uses the command processor, and is much less readable).

Jacques.

Thanks. I'd meant to work on this sooner but other priorities keep sneaking in. For example, it's likely that adding more Gui control types -- such as Date, Up-down, and ListView -- is something that most users would consider a higher priority.

I'll definitely check out that article when the time comes. Thanks.

Thanks for the link Jacques.

A DLL Version of CMDret is now available here