SendMessage ErrorLevel for x64 different than x86

[jballi]

According to the documentation, the ErrorLevel for the SendMessage command is an unsigned integer, i.e. UInt. For the 32-bit version of AutoHotkey, this is correct. The ErrorLevel value is from 0 to 4294967295. However, for the x64 version of AutoHotkey, I just received a -1 value when expecting 4294967295. The script went into an infinite loop because it was testing for 4294967295 (specifically 0xFFFFFFFF) and got -1 instead. From a 32-bit binary perspective, these are all the same value but from an AutoHotkey test perspective, these are completely different values. It broke my script. I'm sure it will break many other scripts when developers start to convert their code to x64.

[lexikos]

This is an error in the documentation, which was written before the x64 release. SendMessage accepts and returns pointer-sized integers. Since AutoHotkey doesn't support unsigned 64-bit integers, a return value of 0xFFFFFFFFFFFFFFFF would produce -1. (I think this would most often come from messages which actually return (LRESULT)-1.)

Technically, LRESULT (the return type of SendMessage()) is a signed type. I don't know why unsigned return values were chosen for AutoHotkey.

[jballi]

I don't know why unsigned return values were chosen for AutoHotkey.

If I were to guess, I imagine it is because it allows for twice as many positive values (2,147,483,648 to 4,294,967,295) without any conversion and most of the return values from the SendMessage command are unsigned integers. I guess it was really because without a return type, it had to be something and "UInt" was as good as anything else.

Your argument is a good one. The x64 version of the SendMessage command needs to be able to return larger values than "UInt". Without the ability to use unsigned 64-bit integers, we get signed 64-bit integer values. The problem I have is that there is a sh*#load amount of code out there that written to expect an unsigned 32-bit value. I reported this problem when I found the inconsistency in one of my functions. I have since found the problem in a half-dozen more functions. I expect to find quite a few more before it is over.

One option is to change the code so that is is backwards compatible to all the code that is out there already, i.e. always return a 32-bit unsigned value. And then create a new command, something like SendMsg, that would have an optional "return type" where the developer could specify an expected return type. If not specified, the return type might be Int for x86 and Int64 for x64. Not sure 'bout the default return type. Just an idea.

Thank you for your consideration.

[just me]

SendMessage: ErrorLevel is set to the word FAIL if there was a problem or the command timed out. Otherwise, it is set to the numeric result of the message, which might sometimes be a "reply" depending on the nature of the message and its target window. This result is an integer between 0 and 4294967295. If the result is intended to be a signed integer, a negative number can be revealed by following this example:

MsgReply := ErrorLevel > 0x7FFFFFFF ? -(~ErrorLevel) - 1 : ErrorLevel

True for 32-bit AHK.

[lexikos]

Your argument is a good one.

Argument? I was simply stating facts.

Now I'm arguing.

One option is to change the code so that is is backwards compatible to all the code that is out there already,

You mean all of the code which is not 64-bit compatible. The x64 build has been out for over 4 years now. What you are suggesting is a sort of half-measure which fixes one specific issue without necessarily making scripts 64-bit compatible, and while likely breaking a whole lot of scripts. All scripts which do not care about the specific value, such as when a handle or pointer is returned, should work regardless of sign or range. If SendMessage was to truncate the value to 32-bit, it would break every script which happens to receive a pointer or handle outside the 32-bit range.

The problem I have is that there is a sh*#load amount of code out there that written to expect an unsigned 32-bit value.

How would such code work on a 64-bit build anyway? Just run it on 32-bit.

The real problem is that those scripts are fundamentally non-portable, and you're trying to use them on a platform they weren't designed for. If the message actually returned 0xFFFFFFFF, the script would still work, because the return value would be 0xFFFFFFFF, not -1. The message is effectively returning (DWORD_PTR)-1, which is 0xFFFFFFFF on 32-bit and 0xFFFFFFFFFFFFFFFF on 64-bit. The scripts would also work if they had converted the value to a signed integer before comparison; even using the method shown in the documentation since v1.0.48.05 or earlier:

Code: Select all

MsgReply := ErrorLevel > 0x7FFFFFFF ? -(~ErrorLevel) - 1 : ErrorLevel

This only affects integers in the range 0x80000000 to 0xFFFFFFFF, due to the way ~ works in AutoHotkey.

If I were to guess, I imagine it is because it allows for twice as many positive values (2,147,483,648 to 4,294,967,295) without any conversion and most of the return values from the SendMessage command are unsigned integers.

I don't believe that positive values in that range are more common than small negative values like -1; excluding handles, where the sign is mostly irrelevant (and truncation to 32-bit could be disastrous).

There are a lot of messages which have no return value (or always return zero), or have "zero" and "non-zero" return values, or return TRUE or FALSE. In those cases, whether the return type is signed or unsigned doesn't matter. Some other messages could in theory return values in that range (e.g. as number of characters in a string) but in practice rarely or never will.

I guess it was really because without a return type, it had to be something and "UInt" was as good as anything else.

Actually, I've had a look at the source code and I think it's probably a consequence of AutoHotkey using SendMessageTimeout, which passes the message result via a DWORD (now DWORD_PTR) output parameter. We can blame Microsoft for the inconsistency between SendMessage and SendMessageTimeout.

True for 32-bit AHK.

That is the exact documentation I was referring to when I said there was an error.

[jballi]

I'm just talking about one command that works one way on the 32-bit build and another way on the 64-bit build. It doesn't matter that the 64-bit build has been out for 4 years or that the documentation has been "wrong" for 4 years, it just matters that it works differently. If you won't change the code to make it compatible with the 32-bit build and/or create a new SendMessage command that accepts a return type, please update the documentation at your convenience.

Thank you for your consideration.

[just me]

We can blame Microsoft for the inconsistency between SendMessage and SendMessageTimeout.

How could Microsoft know, that AHK will use the returned value as the low part of a 64-bit integer in a 32-bit environment?

[lexikos]

I will, of course, update the documentation, and would need to even if the behaviour was changed.

I'm just talking about one command that works one way on the 32-bit build and another way on the 64-bit build.

So you're not talking about anything else that you've brought up in this thread, like your scripts that broke, or your suggestion to change the behaviour? I don't see your point.

It doesn't matter that the 64-bit build has been out for 4 years [...]

In theory, the number of scripts relying on the current behaviour is proportional to the amount of time that behaviour has been in place. The more scripts potentially broken by your suggested change, the more reason there is to reject it. As for your earlier statement; "I'm sure it will break many other scripts when developers start to convert their code to x64." I think that 4 years is long enough for developers to start to convert their code.

How could Microsoft know, that AHK will use the returned value as the low part of a 64-bit integer in a 32-bit environment?

How can Microsoft not be responsible for the inconsistency between two functions that they designed?

[jballi]

I'm just talking about one command that works one way on the 32-bit build and another way on the 64-bit build.

So you're not talking about anything else that you've brought up in this thread, like your scripts that broke, or your suggestion to change the behaviour? I don't see your point.

Your conversion was leading to a discussion about what it takes to convert to 64-bit, why not just run on 32-bit, portablity, etc., etc. I just wanted to return the conversation back to the original problem.

It doesn't matter that the 64-bit build has been out for 4 years [...]

In theory, the number of scripts relying on the current behaviour is proportional to the amount of time that behaviour has been in place. The more scripts potentially broken by your suggested change, the more reason there is to reject it. As for your earlier statement; "I'm sure it will break many other scripts when developers start to convert their code to x64." I think that 4 years is long enough for developers to start to convert their code.

I am a perfect example of why conversion takes a long while, sometime years. I just got a 64-bit computer last year. I just started to convert a project to 64-bit last week. I just ran into this problem a few days ago. My argument is that if anyone was doing any serious 64-bit conversion and they were using the SendMessage command, they would have run into this problem a long long time ago.


Possible update to the documentation. Use or ignore at your discretion.

SendMessage: ErrorLevel is set to the word FAIL if there was a problem or the command timed out. Otherwise, it is set to the numeric result of the message, which might sometimes be a "reply" depending on the nature of the message and its target window.

When ErrorLevel is numeric, the possible values depend on the version of AutoHotkey that is running. When using the 32-bit version of AutoHotkey, the result is a 32-bit unsigned integer value between 0 and 4294967295. When using the 64-bit version of AutoHotkey, the result is an 64-bit signed integer value between -9223372036854775808 and 9223372036854775808.

If the result is intended to be a 32-bit signed integer (a value from -2147483648 to 2147483648) and using the 32-bit version of AutoHotkey, the result can be converted to a signed integer value using the following example:

Code: Select all

MsgReply:=ErrorLevel<<32>>32

If writing for the both the 32-bit and 64-bit version of AutoHotkey, always converting the result to a signed integer might make sense if expecting a 32-bit signed integer value. For example, this test/conversion will work on both the 32-bit and 64-bit version of AutoHotkey:

Code: Select all

if (ErrorLevel<<32>>32=-1)
    {
    ;-- Do stuff here when -1 is returned
    }

[just me]

How can Microsoft not be responsible for the inconsistency between two functions that they designed?

SendMessage function
Return value
Type: LRESULT (typedef LONG_PTR LRESULT)
The return value specifies the result of the message processing; it depends on the message sent.
#if defined(_WIN64)
typedef __int64 LONG_PTR;
#else
typedef long LONG_PTR;
#endif

SendMessageTimeout function
lpdwResult [out, optional]
Type: PDWORD_PTR (typedef DWORD_PTR *PDWORD_PTR;)
The result of the message processing. The value of this parameter depends on the message that is specified.
typedef ULONG_PTR DWORD_PTR;
#if defined(_WIN64)
typedef unsigned __int64 ULONG_PTR;
#else
typedef unsigned long ULONG_PTR;
#endif

What do you think about the type of the value that should be stored in ErrorLevel?

[lexikos]

Your conversion was leading to a discussion about what it takes to convert to 64-bit, why not just run on 32-bit, portablity, etc., etc. I just wanted to return the conversation back to the original problem.

The original problem is that your scripts broke, because without that, the inconsistency of the command is not a problem at all. My feeling (justified or not) was that you changed the topic because you couldn't address any of my arguments. If you've had enough of the discussion that's fine, but consider that there might be a better solution (than just documenting the problem) which we haven't found yet.

Possible update to the documentation.

Thanks.

If the result is intended to be a 32-bit signed integer (...) and using the 32-bit version of AutoHotkey,

It can be used regardless of the version. Actually, I think that if a 64-bit program sends a message to a 32-bit program, the reply is zero-extended (i.e. -1 becomes 0xFFFFFFFF). So the conversion is necessary even on 64-bit if you might be dealing with a 32-bit window. I don't know how that's supposed to work in C/C++; I suppose that programs working across that boundary are rare, and would have to either detect the "bitness" of the window or always truncate to 32-bit for signed values.

What do you think about the type of the value that should be stored in ErrorLevel?

For v1 and with the limitations of AutoHotkey, "should be" is irrelevant. For v2 with current limitations, perhaps it should be always signed, for consistency/portability between 32-bit and 64-bit. However, that still wouldn't solve the problem I mentioned above where the target window is 32-bit and the sender is 64-bit.

Being always unsigned might be more consistent overall, but isn't an option until AutoHotkey supports unsigned 64-bit. I don't think that's going to happen.

[jballi]

If the result is intended to be a 32-bit signed integer (...) and using the 32-bit version of AutoHotkey,

It can be used regardless of the version. Actually, I think that if a 64-bit program sends a message to a 32-bit program, the reply is zero-extended (i.e. -1 becomes 0xFFFFFFFF). So the conversion is necessary even on 64-bit if you might be dealing with a 32-bit window. I don't know how that's supposed to work in C/C++; I suppose that programs working across that boundary are rare, and would have to either detect the "bitness" of the window or always truncate to 32-bit for signed values.

Specifically...

where "a 64-bit program sends a message to a 32-bit program, the reply is zero-extended (i.e. -1 becomes 0xFFFFFFFF)?"

Interesting. Do you have an example where this occurs?

I only ask because I don't have any personal experience with this (yet) but I have a itchy feeling in my gut that this never happens in real life. I only think that because the first rule of fight club... I'm sorry, the first rule of 64-bit emulation is to always convert 32-bit numbers to 64-bit numbers. If the return type is Int64, won't the number be converted correctly?

It this does occur or is a possibility, then yes, definitely yes, this need to be documented. Without some sort of conversion in these cases, all negative return values would be incorrect.

[lexikos]

Strictly speaking, it's impossible for a 32-bit value to be returned by a function which is declared as returning a 64-bit type. Like you said, the value must be converted somehow. There are two types of conversion: zero-extending, used for unsigned numbers, and sign-extending, used for signed numbers. What I'm saying is that Windows (or Wow64) uses zero-extending. Therefore, I suppose that it happens every time a 32-bit program returns a message to a 64-bit program.

I can't find any relevant documentation, so this is according to my testing. I suppose it's technically possible that some messages that the system knows return signed values could be sign-extended rather than zero-extended, but I would not expect it. For custom messages, there's no way that the system could know which method is required.

So if my assumptions are correct, the upper 32 bits are always zero, including the sign bit. This only matters if the number is intended to be signed, the result is negative, and the script doesn't do something like ErrorLevel << 32 >> 32.

You can test it by having a 32-bit script with OnMessage return a negative value and a 64-bit script call it via SendMessage (calling it via DllCall will show that it's not due to the SendMessage command). I also tested it with C#.


I have updated the (online) documentation for SendMessage's ErrorLevel and parameters.

[jballi]

I have updated the (online) documentation for SendMessage's ErrorLevel and parameters.

Looks good. Thanks for the update.

[just me]

You can test it by having a 32-bit script with OnMessage return a negative value and a 64-bit script call it via SendMessage (calling it via DllCall will show that it's not due to the SendMessage command).

Confirmed! Interestingly, if a 32-bit script is sending a message to a 64-bit script, DllCall returns -1 whereas SendMessage returns 4294967295.

IMO, the problem is related only to messages sent between different applications/processes. It should be feasible to check the bitness of the receiving process, but it might cost too much overhead.

On the other hand, messages sent to a window owned by the running AHK process should not cause any problems. They even might not need to be send per SendMessageTimeout. Maybe we could introduce a separate command like GuiSendMsg, GuiCtrlSendMsg, or similar?

BTW: Can it happen that a 64-bit process will return a 64-bit pointer to a 32-bit process which will be truncated?

[lexikos]

Confirmed! Interestingly, if a 32-bit script is sending a message to a 64-bit script, DllCall returns -1 whereas SendMessage returns 4294967295.

As documented, SendMessage returns a number between 0 and 4294967295 on 32-bit. The message result is 32-bit, so 0xFFFFFFFF and -1 have the same bit representation. DllCall returned -1 because you told it to interpret the result as a signed int (or you omitted the return type, which is the same as specifying "int").

Say the 64-bit script returns -1, which is 0xFFFFFFFFFFFFFFFF; converting from 64-bit to 32-bit will always just truncate the top 32 bits, because there's nothing else that can be done. So returning -1 is the same as returning 0xFFFFFFFF.

It should be feasible to check the bitness of the receiving process, but it might cost too much overhead.

It is not possible to know whether the other process intended the value to be signed or unsigned.

BTW: Can it happen that a 64-bit process will return a 64-bit pointer to a 32-bit process which will be truncated?

Every 64-bit value that is returned to a 32-bit process is truncated (but sometimes the top 32 bits are unimportant). In practice, messages accept a pointer to write to; they do not allocate memory and then return it -- what would happen if the message was lost? Pointers aren't valid across process boundaries anyway.

ControlGet deals with this problem for ListViews. It uses VirtualAllocEx to allocate the memory in the target process, and it works. So I see two possibilities:

• When called from a 32-bit process, VirtualAllocEx allocates memory at an address which can be safely truncated to 32-bit. This is likely given the lengths Microsoft went to for interoperability (see below), although this behaviour of VirtualAllocEx is not documented as far as I've found.
• VirtualAllocEx usually returns an address which can be safely truncated to 32-bit, or truncating the address produces a usable address anyway (but not the right one), and the command works most of the time by sheer coincidence.

However, certain types of handles can always be passed back and forward without risk.

64-bit versions of Windows use 32-bit handles for interoperability. When sharing a handle between 32-bit and 64-bit applications, only the lower 32 bits are significant, so it is safe to truncate the handle (when passing it from 64-bit to 32-bit) or sign-extend the handle (when passing it from 32-bit to 64-bit). Handles that can be shared include handles to user objects such as windows (HWND), handles to GDI objects such as pens and brushes (HBRUSH and HPEN), and handles to named objects such as mutexes, semaphores, and file handles.
Source: Interprocess Communication Between 32-bit and 64-bit Applications

[just me]

I wasn't clear enough, with 'truncated' I meant 'corrupted by truncating'.
As an example:
SendMessage, WM_SETTEXT, 0, &TEXT, , ahk_id %HWND
When sent from 64-bit to 32-bit, will the 64-bit pointer be valid after truncation in any case?

It is not possible to know whether the other process intended the value to be signed or unsigned.

The return value of SendMessage() is defined as LRESULT, so it reasonably should be signed.

[lexikos]

Obviously, it depends on whether the value is the same after truncation.

The return value of SendMessage() is defined as LRESULT, so it reasonably should be signed.

That has nothing to do with the other process' intention or meaning of the message result. Otherwise;
The message result return by SendMessageTimeout() is defined as DWORD_PTR, so it reasonably should be unsigned.
The system zero-extends rather than sign-extends results passing from 32-bit to 64-bit, so it reasonably should be unsigned.

[just me]

Obviously, it depends on whether the value is the same after truncation.

So WM_SETTEXT shouldn't be sent from a 64-bit to a 32-bit process because it might cause a crash (access violation)?

That has nothing to do with the other process' intention or meaning of the message result.

I see, it's my lack of knowledge about C/C++. I thought that LRESULT would represent a signed value in this languages.

[lexikos]

WM_SETTEXT is a system message, and as such, the system marshals memory between the two processes. If this wasn't the case, you wouldn't be able to use it at all without VirtualAllocEx/WriteProcessMemory.

I did not say that LRESULT is not signed, I said that it is irrelevant. The process' intention and meaning of the message result depend on the message being sent, not on the return type enforced by the compiler for all messages, which is more often than not bypassed using a type-cast.