Allow AutoHotkey to Handle Binary Data

Propose new features and changes
SOTE
Posts: 1426
Joined: 15 Jun 2015, 06:21

Re: Allow AutoHotkey to Handle Binary Data

26 Feb 2019, 02:06

Excuse me for interjecting, as I know this is a touchy subject. But isn't AHK v2 still Alpha, thus has not achieved stability nor release candidate status. A stable AHK v2 is still a bit off into the future. As AHK_L is the stable version, and what many or the majority are using and relying on it, shouldn't it get necessary attention?

I thought the plan was to introduce stable or beneficial features from AHK v2 into AHK_L or into both, so that the two versions do a slow merge. Where the difference between AHK v2 and AHK_L became small enough that the jump to the newer version would not be painful. At this time, AHK_L is still arguably easier to use and more user friendly, because of it's more easy to understand legacy syntax and widespread use.

And to be more specific to the topic. nnnik, I thought your idea of adding a new feature to address the problem is valid, and which Drugwash and jeeswg look to be in agreement with that part. I don't see a conflict if that is done, because it's giving the same functionality in both (though in a different way). Whenever AHK v2 becomes more stable, people can have a help file identifying how to do the task in the v2 way. Differences on how the new feature for AHK_L would look are understandable, but hopefully what's best for the community will prevail. AHK_L is still the stable official release.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

26 Feb 2019, 04:53

Any work spend on AHK_L is time that AHK v2 will be delayed.
No it was never the plan to introduce all the new features from AHK v2 - some just happened to be compatible and lexikos made them available.
From what I have seen he will not try to make things actively compatible or consider the AHK v1 source when it comes to the addition of new features - so most new features are incompatible.
Recommends AHK Studio
Helgef
Posts: 4709
Joined: 17 Jul 2016, 01:02
Contact:

Re: Allow AutoHotkey to Handle Binary Data

26 Feb 2019, 05:20

Drugwash wrote:We have to keep in mind that there are many good old scripts whose authors are not around anymore and it would be quite a shame to break them by unconditionally pushing people towards v2, seeing that version has quite a lot of breaking changes.
Authors should maintain their own scripts, the program shouldn't stop evolving just because someone wrote a good script. If their scripts are good, they'll run on the old versions. No one will ever be required to use v2 if they do not like. Same is true for v1.
jeeswg wrote:@Helgef: It was important to state *explicitly* that := does not cause truncation (in response to Drugwash). (nnnik did not state this explicitly.)
The op states,
when binary data containing a null character is copied by the assignment (:=) operator, passed to a function as a parameter, or returned from a function, any data after the null is lost
, Drugwash stated that data gets truncated and nnnik showed an example where it is truncated, an example which is supported by the documentation. Then you make one test script and decide everyone is wrong, well guess what, it is you that is wrong. The current implementation will copy the contents in the variable's string buffer up to the stored length. This is not documented and you cannot rely on it, the documentation explicitly says, as already quoted by nnnik,
When a variable's address (e.g. &MyVar) is passed to a function and that function alters the length of the variable's contents, subsequent uses of the variable may behave incorrectly.
This is what your example relies on, you didn't update the string length, hence the := behaves incorrectly, and copies more data than necessary. So no, it was not important to state *explicitly* that := does not cause truncation, because that is a false statement. I have told you several times before to not make these test scripts and draw general conclusions from them, you are misleading yourself and others, this is not a successful approach.
Cheers.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Allow AutoHotkey to Handle Binary Data

26 Feb 2019, 12:24

- @Helgef: var2 := var1 does not cause truncation (any trailing data beyond any null characters is maintained up to var1's stored length, this is undocumented, although arguably implied by the documentation). Any trailing data is kept until you do VarSetCapacity(var, -1) which causes truncation. Agree or disagree?
- (I always advise people to avoid using undocumented features, I only investigated := because Drugwash mentioned it. I already pointed out VarSetCapacity and memmove as a reliable alternative.)
- (A test is an example, not a proof, this is obvious. The word 'test' itself implies uncertainty. I tested a general conclusion derived from the documentation, and made no particular or strong claims.)
- Your last post was good in places but then unclear when it really mattered e.g. you used 'so' but A did not imply B. I would recommend that you edit it if you want an effective response. Also, one self-contained summary post per thread is often ideal, rather than your currently scattered comments, to add clarity and avoid confusion.
- (And if you want to be especially constructive, cite the source code to explain any behaviours, and propose what any future functionality should look like.)

- @nnnik: I liked your AHK v2 comments, but they (and the replies) might be better placed in a new thread. And, be honest, if I had written your post, would you have already moved it to a new thread?
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
Helgef
Posts: 4709
Joined: 17 Jul 2016, 01:02
Contact:

Re: Allow AutoHotkey to Handle Binary Data

27 Feb 2019, 08:18

this is undocumented, although arguably implied by the documentation)
It can't be both, the behaviour of var2 := var1 is undefined when the address of var1 (eg &var1) is passed to a function and that function alters the length of the variable's contents, i.e., it is not documented and there is no implication what the behaviour might be. Hence claiming that it doesn't cause truncation is misleading.
Any trailing data is kept until you do VarSetCapacity(var, -1) which causes truncation. Agree or disagree?
varsetcapacity x, -1 updates the internally stored string length, it doesn't alter the capacity or the content of the variable's string buffer (possibly, disregarding the case where x only contains a cached number, i.e, it wasn't initialised with a string or varsetcapacity).

Cheers.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Allow AutoHotkey to Handle Binary Data

27 Feb 2019, 21:14

- @Helgef: Could you clarify? When you do var2 := var1, functions are not involved.
DllCall() - Syntax & Usage | AutoHotkey
https://autohotkey.com/docs/commands/DllCall.htm
When a variable's address (e.g. &MyVar) is passed to a function and that function alters the length of the variable's contents, subsequent uses of the variable may behave incorrectly. To fix this, do one of the following: 1) Pass MyVar as a "Str" argument rather than as a Ptr/address; 2) [v1.0.44.03+]: Call VarSetCapacity(MyVar, -1) to update the variable's internally-stored length after calling DllCall.
- Something must determine a variable's size (protected data) and capacity (size + surplus unprotected data). It is implied that the internally-stored length determines the variable's size. It is implied that any null characters, inserted via NumPut into the variable, do not affect the internally-stored length (size). (There may have been further documentation evidence.)
- Thus if var1 has the desired internally-stored length (size), so will var2, when you do var2 := var1. [Undocumented but implied.]
- I could have said: 'Any trailing data is *guaranteed/protected* until you do VarSetCapacity(var, -1) which causes truncation'. After VarSetCapacity(var, -1), any trailing data is subsequently not guaranteed to stay the same. Cheers.

- There's also this:
Saving ClipboardAll to a nested object does not work - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=41927&p=191776#p191776
v1 does not support binary data except in very limited cases, such as assigning one variable to another variable (not an array element).
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
User avatar
Raccoon
Posts: 53
Joined: 26 Jul 2014, 16:15

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 09:06

nnnik wrote:
25 Feb 2019, 10:21
For example AHK_Basic didn't have an actual ptr type and was incompatible with 64 bit. Yet we moved to 64 bit and those old scripts broke. Same goes for Ansi/Unicode and similar.
There's a distinct difference between script breakage due to pointers and other platform changes, versus simply deciding to get rid of an entire style of programming because a few devs decided they don't like it and it should go away.

I won't switch to v2 because I actually prefer interactions between command AHK and expression AHK. v2 removes a plethora of programming style that needs to be brought back into the fold in order for v2 to be usable again.

Code: Select all

MyVar = Naked Text

MyVar =
(
I love this style, 
I will not code without it. 
You are destroying simplicity
in the name of purity and are
putting unnecessary an burden on
simple old ladies and secretaries
who would learn from this easy style.
)
I refuse to use v2 out of protest. Fix it.

I would be happy if all functions could be used as commands, and vise-versa.

But a no-command function-only evaluation-only quoted-text environment is rubbish.

These views were made known 10 years ago.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 09:22

Well stick with v1 then.
Recommends AHK Studio
User avatar
Raccoon
Posts: 53
Joined: 26 Jul 2014, 16:15

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 09:24

Sticking to v1, and politely asking that v2 only be treated as a test-bed for new features to be introduced for v1, such as pointers and unicode support. Even if there's an off-chance it might break some things for someone somewhere long forgotten about.

AHK v2 isn't like switching from Python 2 to 3. It's like switching from Python to C-Sharp, or Java.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 09:53

Yeah obviously. AHK went from being a dynamically, loosely typed scripting language to a compiled, statically typed, OOP only language in v2.
You are clearly not exaggerating.
Recommends AHK Studio
SOTE
Posts: 1426
Joined: 15 Jun 2015, 06:21

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 10:55

Raccoon wrote:
01 Mar 2019, 09:06
nnnik wrote:
25 Feb 2019, 10:21
For example AHK_Basic didn't have an actual ptr type and was incompatible with 64 bit. Yet we moved to 64 bit and those old scripts broke. Same goes for Ansi/Unicode and similar.
There's a distinct difference between script breakage due to pointers and other platform changes, versus simply deciding to get rid of an entire style of programming because a few devs decided they don't like it and it should go away.

I won't switch to v2 because I actually prefer interactions between command AHK and expression AHK. v2 removes a plethora of programming style that needs to be brought back into the fold in order for v2 to be usable again.
I think a lot of people agree with you. I prefer v1 myself, though some of my reasons might differ.
Sticking to v1, and politely asking that v2 only be treated as a test-bed for new features to be introduced for v1, such as pointers and unicode support. Even if there's an off-chance it might break some things for someone somewhere long forgotten about.

AHK v2 isn't like switching from Python 2 to 3. It's like switching from Python to C-Sharp, or Java.
I actually think that's a good idea, where v2 is the bleeding edge test-bed for testing features, and what turns out to be actually good ideas are then introduced in v1. At the very least, I do agree with the concept of "slow merging" of v1 and v2, and healthy discussion on what is best.

v2 does have a "different" vibe to it I wouldn't say it's a different language, but it's definitely less user/newbie friendly. The concern with that, is one of the major points of AutoHotkey being lost. If people want to do things the hard way or make life more difficult, there is already C++ waiting for them. I kind of always thought part of the point was to consistently find the easier way to do things in contrast to C++ or even what is viewed as the standard programming practice.

“Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains.” - Steve Jobs

“The definition of genius is taking the complex and making it simple.” - Albert Einstein
User avatar
Drugwash
Posts: 850
Joined: 29 May 2014, 21:07
Location: Ploieşti, Romania
Contact:

Re: Allow AutoHotkey to Handle Binary Data

01 Mar 2019, 14:14

Could we get back on track? Safe and easy handling of binary data in AHK v1.1+.
Oh and I agree with SOTE: KISS.
Part of my AHK work can be found here.
User avatar
jeeswg
Posts: 6902
Joined: 19 Dec 2016, 01:58
Location: UK

Re: Allow AutoHotkey to Handle Binary Data

02 Mar 2019, 11:21

@nnnik:
(1) When do you expect AHK v2 to be released?
Here's an estimate:
Spoiler

(2) Should users just sit and wait, or should they be proactive somehow?

(3) Have you ever maintained 2 versions of the same code?

(4) Should the following Loop (expression) and LoopXXX syntax be added to AHK v1 and v2? Would this be valuable?
why was LoopParse (no space) removed? - Page 4 - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=37&t=41821&p=235007#p235007

(5) Do you still want to remove NumGet/NumPut? Thanks.

[Btw @mods/admins please consider moving the general AHK v2 posts in this thread to a new thread. Thanks. We currently have 2 threads in 1, the 2nd thread started here.]
homepage | tutorials | wish list | fun threads | donate
WARNING: copy your posts/messages before hitting Submit as you may lose them due to CAPTCHA
lexikos
Posts: 9559
Joined: 30 Sep 2013, 04:07
Contact:

Re: Allow AutoHotkey to Handle Binary Data

16 Mar 2019, 05:58

Raccoon wrote:
01 Mar 2019, 09:06
I refuse to use v2 out of protest. Fix it.
I was going to just lurk, having not opened the forum for months and thus far finding today's visit very uninspiring. Then I read this, and had to log in to say thanks! For the laughs. :lolno:


v2 has better support for binary data. nnnik was close enough;
[...] possible in v2 because lexikos removed a ton of technical debt and overhauled most of the behavior.
My requirement for adding new features to both v1 and v2 or back-porting from v2 to v1 is that it requires only trivial changes between the two code bases, or that someone else does all of the work.

I would prefer not to add functions for dealing with binary data unless (or until) I am reasonably sure that they are also what is best for v2.
iseahound
Posts: 1434
Joined: 13 Aug 2016, 21:04
Contact:

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 15:04

Is it too hard to handle the binary data manually? Binary data comes in two forms managed and un-managed. If you are using un-managed code there is no reason to complain about lack of binary support, use the low-level functions. If you are using managed code then stick with the managed framework.

What even is the problem here? Inexperienced users needing super user powers?

Here's a function that compares two pointers to a bitmap and checks to see if the image data match.

Code: Select all

      isBitmapEqual(pBitmap1, pBitmap2) {
         ; Make sure both Bitmaps are valid pointers.
         if !(pBitmap1 && pBitmap2)
            return false

         ; Check if pointers are identical.
         if (pBitmap1 == pBitmap2)
            return true

         pBitmap1_width  := Gdip_GetImageWidth(pBitmap1)
         pBitmap1_height := Gdip_GetImageHeight(pBitmap1)
         pBitmap2_width  := Gdip_GetImageWidth(pBitmap2)
         pBitmap2_height := Gdip_GetImageHeight(pBitmap2)

         ; Match image dimensions. De Morgan's rules!
         if !(pBitmap1_width == pBitmap2_width && pBitmap1_height == pBitmap2_height)
            return false

         ; Find smaller width and height to match bytes.
         ; Sort of unnecessary due to the above statement, but nice to have.
         width := (pBitmap1_width < pBitmap2_width) ? pBitmap1_width : pBitmap2_width
         height := (pBitmap1_height < pBitmap2_height) ? pBitmap1_height : pBitmap2_height
         E1 := Gdip_LockBits(pBitmap1, 0, 0, width, height, Stride1, Scan01, BitmapData1)
         E2 := Gdip_LockBits(pBitmap2, 0, 0, width, height, Stride2, Scan02, BitmapData2)

         ; RtlCompareMemory preforms an unsafe comparison stopping at the first different byte.
         size := width * height * 4  ; ARGB = 4 bytes
         byte := DllCall("ntdll\RtlCompareMemory", "ptr", Scan01+0, "ptr", Scan02+0, "uint", size)

         Gdip_UnlockBits(pBitmap1, BitmapData1)
         Gdip_UnlockBits(pBitmap2, BitmapData2)
         return (byte == size) ? true : false
      }
Here's a function that is simplified from the above, comparing the binary contents of two AutoHotkey variables.

Code: Select all

; Pass the address of the variable. &var or Ptr := Object.GetAddress(Key)
isBinaryEqual(bin1, bin2, size) {
   ; Check if pointers are valid.
   if !(bin1 && bin2)
      return false

   ; Check if pointers are identical.
   if (bin1 == bin2)
      return true

   byte := DllCall("ntdll\RtlCompareMemory", "ptr", bin1, "ptr", bin2, "uint", size)

   return (byte == size) ? true : false
}
EDIT: Fixed IsBinaryEqual()
EDIT 2: ptr fixes.
EDIT 3: Fixed RtlCompareMemory -> ntdll\RtlCompareMemory thanks nnnik.
Last edited by iseahound on 01 Apr 2019, 16:35, edited 3 times in total.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 15:31

Your second code won't work.
Recommends AHK Studio
iseahound
Posts: 1434
Joined: 13 Aug 2016, 21:04
Contact:

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 15:42

Code: Select all

VarSetCapacity(a, 16, 1) ; Fill 16 bytes with the number 1.
VarSetCapacity(b, 16, 1) ; Fill 16 bytes with the number 1.
MsgBox % isBinaryEqual(a, b, 16) ; true (1)

VarSetCapacity(a, 16, 1) ; Fill 16 bytes with the number 1.
VarSetCapacity(b, 16, 2) ; Fill 16 bytes with the number 2.
MsgBox % isBinaryEqual(a, b, 16) ; false (0)

isBinaryEqual(bin1, bin2, size) {
   ; Check if pointers are valid.
   if !(&bin1 && &bin2)
      return false

   ; Check if pointers are identical.
   if (&bin1 == &bin2)
      return true

   byte := DllCall("RtlCompareMemory", "ptr", &bin1, "ptr", &bin2, "uint", size)

   return (byte == size) ? true : false
}
Looks okay to me. I just tested it.

EDIT: Here's a better version I confused bytes and bits :(

Code: Select all

size1 := VarSetCapacity(a, 16, 1) ; Fill 2 bytes with the number 1.
size2 := VarSetCapacity(b, 16, 1) ; Fill 2 bytes with the number 1.
MsgBox % isBinaryEqual(a, b, Max(size1, size2)) ; true (1)

size1 := VarSetCapacity(a, 16, 1) ; Fill 2 bytes with the number 1.
size2 := VarSetCapacity(b, 24, 1) ; Fill 3 bytes with the number 1.
MsgBox % isBinaryEqual(a, b, Max(size1, size2)) ; false (0)

isBinaryEqual(bin1, bin2, size) {
   ; Check if pointers are valid.
   if !(&bin1 && &bin2)
      return false

   ; Check if pointers are identical.
   if (&bin1 == &bin2)
      return true

   byte := DllCall("RtlCompareMemory", "ptr", &bin1, "ptr", &bin2, "uint", size)

   return (byte == size) ? true : false
}
EDIT 2: Everything is wrong here look 2 posts below.
Last edited by iseahound on 01 Apr 2019, 16:13, edited 2 times in total.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 15:46

Yeah you know it's okay to say nothing if you do not understand the problem:

Code: Select all

VarSetCapacity(a, 16, 0) ; Fill 16 bytes with the number 0.
VarSetCapacity(b, 16, 0) ; Fill 16 bytes with the number 0.
NumPut(13,b,1,"UChar") ;set the 2nd bit of b to 13
MsgBox % isBinaryEqual(a, b, 16) ; true (1)

isBinaryEqual(bin1, bin2, size) {
   ; Check if pointers are valid.
	if !(&bin1 && &bin2)
		return false
	
   ; Check if pointers are identical.
	if (&bin1 == &bin2)
		return true
	
	byte := DllCall("RtlCompareMemory", "ptr", &bin1, "ptr", &bin2, "uint", size)
	
	return (byte == size) ? true : false
}
Recommends AHK Studio
iseahound
Posts: 1434
Joined: 13 Aug 2016, 21:04
Contact:

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 16:10

My bad I forgot that AutoHotkey makes copies of the variables when they are delivered to a function and that meant that the null termination was in effect and truncated the string from VarSetCapacity(). The solution was to pass the pointers/addresses directly as I did below, or to use a ByRef variable type.

Code: Select all

VarSetCapacity(a, 16, 0)           ; Fill 16 bytes with the number 0.
VarSetCapacity(b, 16, 0)           ; Fill 16 bytes with the number 0.
NumPut(13, b, 1, "UChar")          ; Set the 2nd bit of b to 13
MsgBox % isBinaryEqual(&a, &b, 16) ; false (0)

isBinaryEqual(bin1, bin2, size) {
   ; Check if pointers are valid.
   if !(bin1 && bin2)
      return false

   ; Check if pointers are identical.
   if (bin1 == bin2)
      return true

   byte := DllCall("RtlCompareMemory", "ptr*", bin1, "ptr*", bin2, "uint", size)

   return (byte == size) ? true : false
}
It's fixed now.
User avatar
nnnik
Posts: 4500
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: Allow AutoHotkey to Handle Binary Data

01 Apr 2019, 16:16

Now the only remaining issue is that I get an ErrorLevel of -4 for any DllCall to RtlCompareMemory.
And yes byref is very restricting so in the end you are essentially back to using a function wrapper around RtlCompareMemory.
:= doesn't work normal comparisons don't work it all doesn't work and you need to remember the workarounds.
Something thats difficult even for long term users after a short break from the topic.
You only delivered a reason as to why these features need an enhancement.
Recommends AHK Studio

Return to “Wish List”

Who is online

Users browsing this forum: No registered users and 34 guests