Returning byte array gets corrupted without ByRef?

Get help with using AutoHotkey (v1.1 and older) and its commands and hotkeys
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Returning byte array gets corrupted without ByRef?

06 Nov 2015, 16:35

Code: Select all


memcpy(ptrAddress, ptrFrom, size) {
	return DllCall("msvcrt\memcpy_s", "Ptr", ptrAddress, "Int", size, "Ptr", ptrFrom, "Int", size, "Int")
}

BaseToDec(n, Base) {
	static U := A_IsUnicode ? "wcstoui64_l" : "strtoui64"
	return, DllCall("msvcrt\_" U, "Str",n, "Uint",0, "Int",Base, "CDECL Int64")
}

DecToBase(n, Base) {
	static U := A_IsUnicode ? "w" : "a"
	VarSetCapacity(S,65,0)
	DllCall("msvcrt\_i64to" U, "Int64",n, "Str",S, "Int",Base)
	return, S
}

HexStringToBufferObject(ByRef buffer, str, repeat := 1) {
	originalStr := str
	Loop, % repeat - 1  {
		str .= " " . originalStr
	}
	hexString := str
	bytes := StrSplit(str, " ")
	VarSetCapacity(buffer, bytes.MaxIndex() + 2, 0)
	for k, hex in bytes {
		val := BaseToDec(hex, 16)
		NumPut(val, buffer, k - 1, "Char")
	}
	StringUpper, hexString, hexString
	
	return { "str" : hexString, "buffer" : &buffer, "size" : bytes.MaxIndex() }
}

GetBufferObjectFrom(ByRef buffer, ptrAddress, bufferSize) {
	VarSetCapacity(buffer, bufferSize, 0)
	memcpy(&buffer, ptrAddress, bufferSize)
	hexString := ""
	Loop % bufferSize {
		val := DecToBase(NumGet(buffer, A_Index - 1, "UChar"), 16)
		if (val < 10) {
			hexString .= "0"
		}
		hexString .= val
		
		if (A_Index != bufferSize) {
			hexString .= " "
		}
	}
	StringUpper, hexString, hexString
	return { "str" : hexString, "buffer" : &buffer, "size" : bufferSize }
}

var1 := HexStringToBufferObject(b, "01 02 03")
var2 := GetBufferObjectFrom(b2, var1.buffer, var1.size)
var3 := GetBufferObjectFrom(b3, var2.buffer, var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str

Above code works, but it's not intuitive, I have to add useless variables b, b2, b3 there.

But it does not work if I remove b, b2, b3 and ByRef buffer variables from the function definitions. Can I get it to work without those? They seem redundant.

I tried also following, and it does not copy the contents, as if the address can't be referenced:

Code: Select all

memcpy(ptrAddress, ptrFrom, size) {
	return DllCall("msvcrt\memcpy_s", "Ptr", ptrAddress, "Int", size, "Ptr", ptrFrom, "Int", size, "Int")
}

BaseToDec(n, Base) {
	static U := A_IsUnicode ? "wcstoui64_l" : "strtoui64"
	return, DllCall("msvcrt\_" U, "Str",n, "Uint",0, "Int",Base, "CDECL Int64")
}

DecToBase(n, Base) {
	static U := A_IsUnicode ? "w" : "a"
	VarSetCapacity(S,65,0)
	DllCall("msvcrt\_i64to" U, "Int64",n, "Str",S, "Int",Base)
	return, S
}

HexStringToBufferObject(str, repeat := 1) {
	originalStr := str
	Loop, % repeat - 1  {
		str .= " " . originalStr
	}
	hexString := str
	bytes := StrSplit(str, " ")
	VarSetCapacity(buffer, bytes.MaxIndex() + 2, 0)
	for k, hex in bytes {
		val := BaseToDec(hex, 16)
		NumPut(val, buffer, k - 1, "Char")
	}
	StringUpper, hexString, hexString
	
	return { "str" : hexString, "buffer" : buffer, "size" : bytes.MaxIndex() }
}

GetBufferObjectFrom(ptrAddress, bufferSize) {
	VarSetCapacity(buffer, bufferSize, 0)
	memcpy(&buffer, ptrAddress, bufferSize)
	hexString := ""
	Loop % bufferSize {
		val := DecToBase(NumGet(buffer, A_Index - 1, "UChar"), 16)
		if (val < 10) {
			hexString .= "0"
		}
		hexString .= val
		
		if (A_Index != bufferSize) {
			hexString .= " "
		}
	}
	StringUpper, hexString, hexString
	return { "str" : hexString, "buffer" : buffer, "size" : bufferSize }
}

var1 := HexStringToBufferObject("01 02 03")
var2 := GetBufferObjectFrom(&var1.buffer, var1.size)
var3 := GetBufferObjectFrom(&var2.buffer, var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
lexikos
Posts: 9593
Joined: 30 Sep 2013, 04:07
Contact:

Re: Returning byte array gets corrupted without ByRef?

06 Nov 2015, 19:09

You are directly referencing the buffers of b, b2 and b3 from the object, by storing &buffer. How exactly would you remove those variables?

If you want the buffer itself (not just its address) to be stored inside the object, use Object.SetCapacity(key, n) and Object.GetAddress(key).

In your second script, buffer is a local variable containing a string. Only up to the first binary zero will be used, because AutoHotkey uses various C-based APIs which take "null-terminated" strings. If you were to use &buffer, the address would become valid after the variable is freed (when the function returns).
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 01:38

@lexikos, Thanks.

The first example works that's why it includes &buffer as return variable. It seemed to be only way I got it working, when having buffer as a local variable, and returning address, this usually causes using dangling pointer in other languages, and I knew it wouldn't be a good thing to do but in case it was reference it didn't happen.

Second example is the sane version, but it didn't work. However, I didn't know I have to use Object.SetCapacity also, it was not intuitive. One would think that when assigning a variable which has already been resized using VarSetCapacity you don't have to deal with it's size anymore.
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 10:51

lexikos wrote:You are directly referencing the buffers of b, b2 and b3 from the object, by storing &buffer. How exactly would you remove those variables?

If you want the buffer itself (not just its address) to be stored inside the object, use Object.SetCapacity(key, n) and Object.GetAddress(key).

In your second script, buffer is a local variable containing a string. Only up to the first binary zero will be used, because AutoHotkey uses various C-based APIs which take "null-terminated" strings. If you were to use &buffer, the address would become valid after the variable is freed (when the function returns).
I think I spoke too soon, it still corrupts data with some inputs e.g.:

Code: Select all

memcpy(ptrAddress, ptrFrom, size) {
	return DllCall("msvcrt\memcpy_s", "Ptr", ptrAddress, "Int", size, "Ptr", ptrFrom, "Int", size, "Int")
}
 
BaseToDec(n, Base) {
	static U := A_IsUnicode ? "wcstoui64_l" : "strtoui64"
	return, DllCall("msvcrt\_" U, "Str",n, "Uint",0, "Int",Base, "CDECL Int64")
}
 
DecToBase(n, Base) {
	static U := A_IsUnicode ? "w" : "a"
	VarSetCapacity(S,65,0)
	DllCall("msvcrt\_i64to" U, "Int64",n, "Str",S, "Int",Base)
	return, S
}
 
HexStringToBufferObject(str, repeat := 1) {
	originalStr := str
	Loop, % repeat - 1  {
		str .= " " . originalStr
	}
	hexString := str
	StringUpper, hexString, hexString
	bytes := StrSplit(str, " ")
	res := { "str" : hexString, "buffer" : 0, "size" : bytes.MaxIndex() }
	res.SetCapacity("buffer", bytes.MaxIndex() + 2)
	VarSetCapacity(buffer, bytes.MaxIndex() + 2, 0)
	VarSetCapacity(val, 1, 0)
	vals := ""
	for k, hex in bytes {
		val := BaseToDec(hex, 16)
		vals .= val . " "
		NumPut(val, buffer, k - 1, "UChar")
	}
	res.vals := vals
	res.buffer := buffer
	; res.ptr := res.GetAddress("buffer") ; for convinience
	return res
}
 
GetBufferObjectFrom(ptrAddress, bufferSize) {
	hexString := ""
	res := { "str" : "", "buffer" : 0, "size" : bufferSize }
	res.SetCapacity("buffer", bufferSize)
	VarSetCapacity(buffer, bufferSize, 0)
	memcpy(&buffer, ptrAddress, bufferSize)
	Loop % bufferSize {
		val := DecToBase(NumGet(buffer, A_Index - 1, "UChar"), 16)
		if (val < 10) {
			hexString .= "0"
		}
		hexString .= val
 
		if (A_Index != bufferSize) {
			hexString .= " "
		}
	}
	StringUpper, hexString, hexString
	res.str := hexString
	res.buffer := buffer
	; res.ptr := res.GetAddress("buffer") ; for convinience
	return res
}
var1 := HexStringToBufferObject("FF 00 00 00 FF 00 FF 00 FF")
var2 := GetBufferObjectFrom(var1.GetAddress("buffer"), var1.size)
var3 := GetBufferObjectFrom(var2.GetAddress("buffer"), var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
But with this input it works:

Code: Select all

var1 := HexStringToBufferObject("FF 00 FF 00 FF 00 FF 00 FF")
var2 := GetBufferObjectFrom(var1.GetAddress("buffer"), var1.size)
var3 := GetBufferObjectFrom(var2.GetAddress("buffer"), var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
I also tried to NumPut directly res.buffer, but it didn't work at all. Grrh.
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 11:27

Okay I found out, this made absolutely no sense to me.

Object.SetCapacity and Object.GetAddress must be called without quotes, meaning it looks just like passing a variable. If you do pass quotes, the GetAddress returns a pointer, but who knows where? SetCapacity returns zero (so I figured this one out).

Secondly NumPut can be called only by GetAddress when using object.

Here is the version I hope works:

Code: Select all

memcpy(ptrAddress, ptrFrom, size) {
	return DllCall("msvcrt\memcpy_s", "Ptr", ptrAddress, "Int", size, "Ptr", ptrFrom, "Int", size, "Int")
}
 
BaseToDec(n, Base) {
	static U := A_IsUnicode ? "wcstoui64_l" : "strtoui64"
	return, DllCall("msvcrt\_" U, "Str",n, "Uint",0, "Int",Base, "CDECL Int64")
}
 
DecToBase(n, Base) {
	static U := A_IsUnicode ? "w" : "a"
	VarSetCapacity(S,65,0)
	DllCall("msvcrt\_i64to" U, "Int64",n, "Str",S, "Int",Base)
	return, S
}
 
HexStringToBufferObject(str, repeat := 1) {
	originalStr := str
	Loop, % repeat - 1  {
		str .= " " . originalStr
	}
	hexString := str
	StringUpper, hexString, hexString
	bytes := StrSplit(str, " ")
	res := { "str" : hexString, "size" : bytes.MaxIndex() }
	res.SetCapacity(buffer, bytes.MaxIndex() + 2)
	for k, hex in bytes {
		NumPut(BaseToDec(hex, 16), res.GetAddress(buffer), k - 1, "UChar")
	}
	return res
}
 
GetBufferObjectFrom(ptrAddress, bufferSize) {
	hexString := ""
	res := { "str" : "", "size" : bufferSize }
	cap := res.SetCapacity(buffer, bufferSize)
	memcpy(res.GetAddress(buffer), ptrAddress, bufferSize)
	Loop % bufferSize {
		val := DecToBase(NumGet(res.GetAddress(buffer), A_Index - 1, "UChar"), 16)
		if (val < 10) {
			hexString .= "0"
		}
		hexString .= val
 
		if (A_Index != bufferSize) {
			hexString .= " "
		}
	}
	StringUpper, hexString, hexString
	res.str := hexString
	return res
}
var1 := HexStringToBufferObject("FF 00 00 00 00 00 FF 00 FF")
var2 := GetBufferObjectFrom(var1.GetAddress(buffer), var1.size)
var3 := GetBufferObjectFrom(var2.GetAddress(buffer), var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
just me
Posts: 9466
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:17

Your last version is using the empty string ("") as a key. Because you pass buffer without quotes it's treated as a variable containing the key, but buffer is empty. Try this one:

Code: Select all

memcpy(ptrAddress, ptrFrom, size) {
	return DllCall("msvcrt\memcpy_s", "Ptr", ptrAddress, "Int", size, "Ptr", ptrFrom, "Int", size, "Int")
}

BaseToDec(n, Base) {
	static U := A_IsUnicode ? "wcstoui64_l" : "strtoui64"
	return, DllCall("msvcrt\_" U, "Str",n, "Uint",0, "Int",Base, "CDECL Int64")
}

DecToBase(n, Base) {
	static U := A_IsUnicode ? "w" : "a"
	VarSetCapacity(S,65,0)
	DllCall("msvcrt\_i64to" U, "Int64",n, "Str",S, "Int",Base)
	return, S
}

HexStringToBufferObject(str, repeat := 1) {
	originalStr := str
	Loop, % repeat - 1  {
		str .= " " . originalStr
	}
	hexString := str
	StringUpper, hexString, hexString
	bytes := StrSplit(str, " ")
	res := {"str": hexString, "buffer": "", "size": bytes.MaxIndex()}
	res.SetCapacity("buffer", bytes.MaxIndex() + 2)
	bufptr := res.GetAddress("buffer")
	vals := ""
	for k, hex in bytes {
		val := BaseToDec(hex, 16)
		vals .= val . " "
		NumPut(val, bufptr + 0, k - 1, "UChar")
	}
	res.vals := vals
	return res
}

GetBufferObjectFrom(ptrAddress, bufferSize) {
	hexString := ""
	res := {"str": "", "buffer": "", "size": bufferSize }
	res.SetCapacity("buffer", bufferSize)
	bufptr := res.GetAddress("buffer")
	memcpy(bufptr, ptrAddress, bufferSize)
	Loop % bufferSize {
		val := DecToBase(NumGet(ptrAddress + 0, A_Index - 1, "UChar"), 16)
		if (val < 10) {
			hexString .= "0"
		}
		hexString .= val

		if (A_Index != bufferSize) {
			hexString .= " "
		}
	}
	StringUpper, hexString, hexString
	res.str := hexString
	return res
}
var1 := HexStringToBufferObject("FF 00 00 00 FF 00 FF 00 FF")
var2 := GetBufferObjectFrom(var1.GetAddress("buffer"), var1.size)
var3 := GetBufferObjectFrom(var2.GetAddress("buffer"), var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
Interestingly, res.SetCapacity("buffer", bufferSize) fails if res.buffer was initialized with a numeric value, 0 in this case, and res.GetAddress("buffer") does not return a valid pointer then.

Code: Select all

Obj := {Buffer: 0}
MsgBox, % Obj.SetCapacity("Buffer", 32)
MsgBox, % Obj.GetAddress("Buffer")

Obj := {Buffer: ""}
MsgBox, % Obj.SetCapacity("Buffer", 32)
MsgBox, % Obj.GetAddress("Buffer")
I'm not sure if this is intended.
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:19

just me wrote:Your last version is using the empty string ("") as a key.
My last version works, 100% confirmed. It makes no sense, but that's the way it is.
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:20

Oh did you read my last post?

SetCapacity and GetAddress must be called without quotes.

obj.GetAddress(buffer) and obj.SetCapacity(buffer, 5)

It's one peculiarity of AHK.
just me
Posts: 9466
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:38

Did you read my last post?
just me
Posts: 9466
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:48

If you still are in doubt, add one statement to your last script:

Code: Select all

...
buffer := "Hello world!" ; <<<<< added
var1 := HexStringToBufferObject("FF 00 00 00 00 00 FF 00 FF")
var2 := GetBufferObjectFrom(var1.GetAddress(buffer), var1.size)
var3 := GetBufferObjectFrom(var2.GetAddress(buffer), var2.size)
MsgBox % var1.str . " == " . var2.str . " == " . var3.str
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 12:49

just me wrote:Did you read my last post?
I read your last post, but it does not work at all. Only thing that works correctly is the one without quotes:

Code: Select all

Obj := {Buffer: 0}
MsgBox, % Obj.SetCapacity(Buffer, 32)
MsgBox, % Obj.GetAddress(Buffer)
 
Obj := {Buffer: ""}
MsgBox, % Obj.SetCapacity(Buffer, 32)
MsgBox, % Obj.GetAddress(Buffer)
This works, not the one with quotes at all.

And also, you can read from it:

Code: Select all

Obj := {Buffer: 0}
MsgBox, % Obj.SetCapacity(Buffer, 15)
MsgBox, % Obj.GetAddress(Buffer)
MsgBox, % Obj.GetCapacity(Buffer)
 
Obj := {Buffer: ""}
MsgBox, % Obj.SetCapacity(Buffer, 28)
MsgBox, % Obj.GetAddress(Buffer)
MsgBox, % Obj.GetCapacity(Buffer)
So the quoted version is not correct, it should throw error in my opinon.

Btw, with "quoted" I don't mean the initialization, it's not required, but quotes around the argument.
lexikos
Posts: 9593
Joined: 30 Sep 2013, 04:07
Contact:

Re: Returning byte array gets corrupted without ByRef?

07 Nov 2015, 17:25

just me wrote:I'm not sure if this is intended.
Yes, it's intended. Expressions, methods and many functions by convention return an empty string on error in v1. Except when truncating, Object.SetCapacity(key, capacity) preserves the data. If the field contained a number or object, it would need to be converted to a string, which would cause loss of information (for a number, the type of value). It's also simpler the current way.
ciantic wrote:Oh did you read my last post?
Did you read the manual?

The first parameter of GetAddress or SetCapacity in its two-parameter mode is the key of a key-value pair, or in other words, the array index. You can pass almost any value you want, including an empty string, as long as you pass the same value each time (to refer to the same key-value pair). If you pass buffer, it uses the contents of that variable. If you pass "buffer", it uses that string. If (buffer = "buffer"), then passing either one will achieve exactly the same result.

In short, just me is correct. As far as I can tell, the script with quotes ("buffer") also works just fine.
If you do pass quotes, the GetAddress returns a pointer, but who knows where?
I do, and so should anyone who reads the manual. Every time GetAddress returns a pointer, it is a pointer to the string buffer of the value (array element) corresponding to the key you specified.
SetCapacity returns zero (so I figured this one out).
If SetCapacity returned zero, that would mean that it set the capacity to zero, which it would only do if you told it to. On failure, it returns an empty string, not zero.

What you wrote is equivalent to this, because as just me said, Buffer is an empty variable:

Code: Select all

Obj := {Buffer: 0}
MsgBox, % Obj.SetCapacity("", 32)
MsgBox, % Obj.GetAddress("")
 
Obj := {Buffer: ""}
MsgBox, % Obj.SetCapacity("", 32)
MsgBox, % Obj.GetAddress("")
Obviously initializing Obj.Buffer will not affect Obj[""].

Now try this:

Code: Select all

Obj := {(Buffer): 0}   ; {(expression): ...} vs {literal: ...}
MsgBox, % Obj.SetCapacity(Buffer, 32)
MsgBox, % Obj.GetAddress(Buffer)
 
Obj := {(Buffer): ""}
MsgBox, % Obj.SetCapacity(Buffer, 32)
MsgBox, % Obj.GetAddress(Buffer)
The first two boxes are blank, showing that it is not valid to use SetCapacity or GetAddress on an array element which already contains a number. If you change (Buffer) back to Buffer and use quotes as posted by just me, you will get the same result.
ciantic
Posts: 19
Joined: 24 Oct 2015, 15:39
Contact:

Re: Returning byte array gets corrupted without ByRef?

08 Nov 2015, 03:44

lexikos wrote:In short, just me is correct.
Yes. I was wrong.

Though my last script worked (miraculously) which made me to think SetCapacity works just like VarSetCapacity, that you can and should pass it uninitialized variable "without quotes".

I have not looked at AHK 2, but I hope it throws more errors when using uninitialized variables like cup cakes.
lexikos
Posts: 9593
Joined: 30 Sep 2013, 04:07
Contact:

Re: Returning byte array gets corrupted without ByRef?

08 Nov 2015, 05:38

ciantic wrote:Though my last script worked (miraculously) which made me to think SetCapacity works just like VarSetCapacity, that you can and should pass it uninitialized variable "without quotes".
I hope you mean that you thought it was miraculous, but now fully understand, because two of us have now explained and demonstrated why your script worked.
I have not looked at AHK 2, but I hope it throws more errors when using uninitialized variables like cup cakes.
Use #Warn if you want that.

There are no truly uninitialized variables in AutoHotkey. All variables are initialized to empty prior to use.

Return to “Ask for Help (v1)”

Who is online

Users browsing this forum: No registered users and 71 guests