AutoHotkey Community

You wrote about returning a pointer, your little code example didn't have ByRef.
Of course, a ByRef parameter is updated, so data "comes out the same way"...

_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

@PhiLho: There are more than one issues:

- AHK copies variable parameters to local variables, not only computed expressions. This does not always waste time, only if you already have your input data in a variable and you don't need it any more (can be overwritten). If an expression is the parameter, it has to be evaluated and stored either explicitly to a variable given ByRef to a function, or implicitly, assigning it to a local parameter, which take the same time.

- When something is returned from a function, it has to be copied to a variable (x:=f(y)) or to a temporary storage (f(y)+1), therefore this extra copy time can be saved with an updated ByRef parameter. Again, this does not save time if you have to manually save that value for further processing (x:=f(y) vs. ByRef: f(y), x:=y). This is not that common. The problem is that the copy induced by the Return from a function takes more than twice the time necessary to copy the data. This is the important observation here.

- In case of a function return, like x:=f(y) AHK could just do some pointer manipulation. The storage to be returned could be allocated in the caller's memory space, and there will be NO data copy necessary. This not done, as you, Titan and I verified.

Let me rephrase it.. I want to get binary data out of a function in one line.

_________________
URLGet - Internet Explorer based Downloader
StartEx - Portable Shortcut Link

Let me rephrase it: Currently, you cannot. "That's a limitation of AHK."
So it looks like a wish, I suppose.

_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Actually, I was answering/supporting Laszlo

_________________
URLGet - Internet Explorer based Downloader
StartEx - Portable Shortcut Link

@Skan,

The inabilty to read and write binary files is unfortunately a problem we all have to suffer, so its better to understand why we can't. As PhiLho says AHK cannot possible read binary files per se. This is not a limitation of AHK but the fundamental design of all string handling , the dreaded NULL terminator.

Back in the early days of MSBasic (when Bill was just another kid on the block), things were handled differently. Now let's recap

memory max 65535 just 2 bytes to hold segment + offset of string pointer
string max 256 just 1 byte to hold the length. Later 65535
That arrangement allowed one to make a string that contained all 256 bytes. You could manipulate string pointers using varptr, and bytes could be read using asc$ and written using chr$. So of course it begs the question: what nut introduced the NULL terminator. And no I don't know.

Below shows proof of concept that AHK DOES in fact handle binary files and strings just as well as text. Of course the subtlety here is we've made sure it doesn't contain any NULL's ie chr(0).



It doesn't mean there's no way out, but for the moment certainly, the easiest way is to treat the numbers as an array of 2byte hexA. I hope this helps a bit.

Regards

Dave.

_________________
Simple ideas lie within reach, only of complex minds

These limitations are well documented in the Help, and solutions for reading and writing binary files (based on dll calls) were already posted a year and a half ago. There you can read: "The binary file I/O functions store/load the data to/from AHK variables, which may contain NULL characters, therefore, they must not be manipulated as ordinary strings. Not even assignments or StrLen is allowed. If these variables are used as function parameters, they must be defined as ByRef, otherwise the strings get truncated at the first NULL".

You have to distinguish STRINGS form other (binary) data. Strings were introduced to handle text, mostly I/O messages. Texts do not contain certain characters, so it is natural to pick one as a delimiter. If you use strings for storing other data, you are in the state of sin.

There is no problem with NUL delimited strings, but only with their misuse. AHK is a string based language, where everything is a string: text, numbers, ID's, addresses, etc. Therefore, AHK cannot handle binary data in its native form. Several posts provided solutions (started almost two years ago) by coding binary data into strings of allowed characters: hex digits, base64, base85 or pebwa (invented by PhiLho).

@Laszlo,

Oh yes Sir, I'm fully aware of all the references, possibly ad nauseum. The fact however is, that whilst we tell people that it's not possible, I thought it might be useful to people that don't have our knowledge, as to why it doesn't work. It also serves as a challenge, because anyone with the correct knowledge and determination, could quite easily write an add-on library that would provide all the necessary functionality.


Regards

Dave.

_________________
Simple ideas lie within reach, only of complex minds

OK.
Well, such feature might come handy when arrays will be implemented: they are often large, and there should be some mechanism to return an array created in a function, without making it global.
Perhaps some special Return telling: push the content of this variable in a global scope (ie. don't erase/overwrite this memory when going out of function scope) and return a reference to it.
Something like:
od is local to G, otherData is local to F but od gets its content without a copy of data.
Limiting this to Return values limits the use of references: they are powerful, but hard to understand (for newbies) and to manage, eg. when doing cross-references (I write about arrays here).

_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Exactly ..



Would not this be nice ? Can you take this to Mr.Chris ?

Thanks for understanding PhiLho, though I sounded vague.

_________________
URLGet - Internet Explorer based Downloader
StartEx - Portable Shortcut Link

My point was that this has to be done always. It does not cause any harm at short return values but saves a lot of time at copying large return values.

This does not solve all the problems with returning binary buffers, because an assignment x := f(y) still truncates the result at the first NUL. Either the assignment in AHK needs to be changed or a special function BinCopy(Byref TO, ByRef FROM, Len=0) has to be used.

It truncates because it makes a copy, the point of my proposal is precisely to avoid such copy.
Obviously a special Return mechanism should use a special assignment behind, otherwise it wouldn't work. Assignments of such value should take note it is a reference/pointer, not a simple value, and ajust the target variable accordingly.

I don't know the current behavior of AHK code, but a possible mechanism is perhaps:
Adjust memory of the variable on the left side of the assignment to the size of the returned string (which is still in the section of the local vars of the function);
Copy the string from the local var to the memory of the variable (that's where the truncate takes place).

The new mechanism can be to free the memory of the assigned variable and makes it point to the memory of the returned value which no longer belong to the list of local vars.

Or, if it clashes too much with current design, we can transform the Return ByRef to a Return BySize...
Ie. the new mechanism could be:
Adjust memory of the variable on the left side of the assignment to the size of the returned variable (that excludes probably using this to return expressions);
Copy the data from the local var to the memory of the variable with a memcpy instead of strpcy (or similar).
There is still a copy in the process, which can be expansive, but to avoid this we can still use ByRef parameters.

_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")

Yes, there are as many as two copies involved when a user-defined function returns a string: first, the function copies its return value to temporary memory, then the caller of the function copies that result into its variable. However, there are some optimizations that avoid one or both of these copies; but currently, the best optimizations apply only to built-in functions like SubStr() and RegExReplace().

Yes, there might be such an optimization. There are so many string/expression optimizations that I can no longer understand them all, and thus resort to comments in the code. I will continue to add more optimizations when the mood occurs, which happens a lot but has the drawback of taking time away from new features.

Originally, one of my main motivations for having ByRef was performance; so until something better comes along, ByRef can be used to "return" huge strings.

Yes, I tried to allow a function that returns a non-static local to surrender that local to its caller (since it's about to be freed anyway). This would save at least one mem-copy. However, the complexity and code size created some problems; so I took it out for now. I'll probably try it again at some point.

That's an interesting idea. It might be possible to directly assign to a variable the temporary storage that holds a return value (rather than making a copy). I've made a note to look into it.

Yes, this seems to achieve the same improvement as the previous paragraph. Hopefully at least one of these approaches will be feasible.

The drawback to optimizations like these is that they tend to create recursion difficulties and interdependencies between sections of the code. This reduces modularity, maintainability, and (slightly) the overall performance of expressions (since extra checks are involved even in trivial expressions that have no big strings). So these factors must be weighed against the benefit.

This is interesting. Although it might be made less important by future optimizations (such as the ones mentioned earlier), it's something to consider for the future.

Thanks for the great ideas.

I understand that Laszlo would prefer a generic solution, but in 90 to 99% of function uses, a Return by copy is OK, as most return values are small.
So if doing optimization is made at the risk of breaking something (recursion for example), I would prefer a special Return to operate on large values, ie. programmer will need to tell AHK: "This data is special, treat it in a special way".
If Chris can make it generic, well that's better, of course.

_________________
vPhiLho := RegExReplace("Philippe Lhoste", "^(\w{3})\w*\s+\b(\w{3})\w*$", "$1$2")