Unexpected behaviour of nested (2D) arrays Topic is solved

[ahketype]

I'm not sure if this is a bug or known issue, or if there's just something I haven't got my head around about arrays (I mean, something else, LOL, there's plenty). Adding 'rows' (i.e. first index arrays) to a parent array to create a 2D array gives odd behaviour depending on how it's done. It's difficult to describe, but the test program below shows it.

Un-commenting the later instances of child := [] fixes it from that point on. With both commented out, the last added version of child overwrites all the way back to the second row, added as child := [4, 5, 6]

Code: Select all

; How does a 2D array differ from one built as sub-arrays added to a parent array?
parent := []
child := []
a := 1
b := 2
c := 3
child[a] := 1
child[b] := 2
child[c] := 3
parent.push(child)
gosub, Output

child := [4, 5, 6]
parent.push(child)
gosub, Output

;~ child := []
child[a] := 7
child[b] := 8
child[c] := 9
parent.push(child)
gosub, Output

;~ child := []
child[a] := 10
child[b] := 11
child[c] := 12
parent.push(child)
gosub, Output

Exit

Output:
out := "Child array`n"
Loop, 3
	out .= child[A_Index] . A_Tab
out .= "`n`nParent array`n"
Loop, % parent.MaxIndex()
{
	i := A_Index
	Loop, 3
		out .= parent[i, A_Index] . A_Tab
	out .= "`n"
}
MsgBox % out
return

[Hardcoder]

Hi, the output from your script is exactly what I expected from the code.
When you assign an array [] to a variable, you are basicly giving the object a name, the variable is just a container.
When you push the child variable in the parent array, you push the object, that is the underlying array, into the other array, not the variable child.

Imagine child as a named box, where you can put stuff in.

So, when you assign something to child, you put it in the "box" child / give that something the name child.
But when you assign something to an array subscript on child, eg. child[1], you are accessing the array that currently is refered to by child, aka has currently the name child.

Your commented lines, that you say "fix" the behaviour, are creating new arrays and assign them to the variable (the name) child.
When you then push child in parent, you are pushing the new array, referred to by child, into parent, not the variable child itself.

I hope that makes any sense

[ahketype]

Hi, the output from your script is exactly what I expected from the code.
When you assign an array [] to a variable, you are basicly giving the object a name, the variable is just a container.
When you push the child variable in the parent array, you push the object, that is the underlying array, into the other array, not the variable child.

You push the contents in, yes.

Imagine child as a named box, where you can put stuff in.

Yep.

So, when you assign something to child, you put it in the "box" child / give that something the name child.
But when you assign something to an array subscript on child, eg. child[1], you are accessing the array that currently is refered to by child, aka has currently the name child.

Well, it's "accessing" (in order to set) a particular indexed value (in the case above, index 1). That's not in dispute.

Your commented lines, that you say "fix" the behaviour, are creating new arrays and assign them to the variable (the name) child.

Yes. Do you get different results when you un-comment those lines? I just realised I assume other people do, and I didn't say what happens in my case (very clearly). Without recreating the child array, the parent array's previous row(s) appear to be changed by the push statement, rather than just the new row added.

When you then push child in parent, you are pushing the new array, referred to by child, into parent, not the variable child itself.

It does indeed push (add to the end of the array, as a new row) those contents. Just if child hasn't been re-created prior to setting its new contents and pushing it, somehow it CHANGES parent (already printed out earlier with so many rows) at rows lower than the row it's adding. If re-created, however, it does what I'd expect, just adds the array child as a new row in parent. Why would pushing [7, 8, 9] into the array [[1, 2, 3], [4, 5, 6]] create the new parent [[1, 2, 3], [7, 8, 9], [7, 8, 9]] ?

I hope that makes any sense

Maybe the above clarifies what's happening to me. If not, maybe I need some output screenshots or something. Thanks anyway!

[Chunjee]

May I suggest a function instead of gosub for viewing your object?

Array_Gui is quite nice for starters: https://gist.github.com/G33kDude/47f97ed521330820e7419591601de01f

I am partial to print() these days: https://gist.github.com/G33kDude/b12aae376d3f9dbdd857e1d92b166dc5

[Hardcoder]

I know exactly what you mean, and what you expect the code to do.
But as I said, the code does exactly what I expect it to / what it is supposed to do.

When you create the variable child, you create the name child as a synonym for a memory address.
For simplicity's sake let's say the memory address is 0x0000, and this is just an address like your home address.
Then child is just an entry in a citizen register:

Code: Select all

variable child pointing to address 0x0000

When you assign an integer value to that variable, its value gets stored at that address, like becoming the resident at that address.

Code: Select all

child := 1

Now, the value 1 is stored at the memory address the variable child is pointing to:

Code: Select all

memory content at address 0x0000 == 1

When you access child, you get the content that is stored at the address it is pointing to, in this case 1.

Now, when you assign an object like an array to that variable, the memory address of that array will be stored in the variable.

Lets say, the array is stored at memory address 0x1000, then child contains that address.
When you now access the arrays contents like

Code: Select all

child[1]

You actually access the memory at 0x1000 plus a value for the index, the compiler/interpreter does this for you, it knows through that syntax

Code: Select all

child[...]

that you try to access memory indirectly through the address that is stored in child.

When you set a value to an array element as in

Code: Select all

child[1] := 123

the value 123 gets stored at 0x1000 (plus the value for the index).

Since you never changed the address that child is pointing at (the array at 0x1000), you still access the same memory.
When you uncomment your "fix", you assign a new array's address (let's say 0x2000) to child.
Now, when you set a value to

Code: Select all

child[1] := 123

you actually change the contents of the memory at 0x2000 (again + index).

I feel like this is a much more complicated description than the first, but maybe it helps.

[Hardcoder]

Yeah, a visualisation will be better than my description.

[Hardcoder]

I added the memory address of child/parent and parents children (first dimension) to your message box.
When you compare the addresses of the parents children (first dimension) to child, you'lll see when they are actually the same.
Every time you run this, the memory addresses will be different (random), that is on purpose from the OS.

Code: Select all

parent := []
child := []
a := 1
b := 2
c := 3
child[a] := 1
child[b] := 2
child[c] := 3
parent.push(child)
gosub, Output

child := [4, 5, 6]
parent.push(child)
gosub, Output

;~ child := []
child[a] := 7
child[b] := 8
child[c] := 9
parent.push(child)
gosub, Output

;~ child := []
child[a] := 10
child[b] := 11
child[c] := 12
parent.push(child)
gosub, Output

Exit

Output:
out := "Child array`t" &child "`n"
Loop, 3
	out .= child[A_Index] . A_Tab
out .= "`n`nParent array`t" &parent "`n"
Loop, % parent.MaxIndex()
{
	i := A_Index
	Loop, 3
		out .= parent[i, A_Index] . A_Tab
	out .= "`t" &parent[i] "`n"
}
MsgBox % out
return

[ahketype]

@Hardcoder, OK, I think I've got it. This is a weird revelation to me. I mean, I've known about variables and addressing since 1983 when I started coding. What I didn't realise is that the push() function in this case (pushing an array into another) only places the address of the pushed array at that index, not actual values. I thought it appended the array by the current values of the given "child" array. So I assumed you must have misunderstood me or the program was working differently on your system.

This is different for simple (non-array) variable contents. If I say x := 4 and array.push(x), I won't come back to find the array containing a new value of x that I might set later. It too is just a 'name' with an address where the value of x is to be found. So, as far as the syntax goes, it seems rather dumb that redefining an array by setting its contents positively (without the specific child := []) doesn't recreate it anew. It's like having to say x := "" before reassigning a different value to x.

Anyway, you seem to have explained the reason, so I guess the thread is solved!

[ahketype]

Follow-up questions - if anyone can help further:

1. Does this pushing of addresses mean that it is more memory intensive to create a 2D array by this method, repeatedly setting a 'row' of values as a separate 1D array and pushing it, since each row then has its own address in memory? If all I want is actually a 2D table, is it better just to set its values with array[row, column] := value?

2. If I do the latter, do I lose the ability to use array.InsertAt(index, ...), because that requires a 'row', and hence an array? It seems I can create a 2D table directly and then use array.RemoveAt(index), with that row removed from the array.

3. Or can I get round the address-storing issue by using array.InsertAt(index, [val1, val2, val3,...])? The values there must be part of a temporary array in memory, presumably, but have no name, so is that memory re-used automatically? The result seems just the same as if I said child := [val1, val2, val3], array.InsertAt(index, child). EDIT: I mean the result is superficially the same, ignoring the whole whether-it's-a-separate-object issue.

And perhaps similarly for array.push([val1, val2, val3,...]) when creating the array row by row. These 'temporary arrays' presumably are not objects.

@Chunjee - thanks, those look great. Print() is particularly helpful for debugging.

[ahketype]

Oh, more testing, and yes, these 'row' arrays pushed as literal data are objects, stored at a given address just as if they had their own variable names (and presumably are stored in the parent array as such). But as they never had a name, I can't accidentally redefine them (e.g., by forgetting to use child := []).

[Hardcoder]

Hi, one thing I want to add to your first reply now, is that in AHK this is even a bit more "inconsistent" compared to static languages, since when passing arrays to a function, by default they are indeed copied.
When you want to change the original array inside of a function, you have to declare the parameter as ByRef.
This is very different from a static language like C or Java, and an important consideration when you work with large arrays.

1. I would assume(!) that when assigning values to a presumed 2D array, AHK would do more or less the same as you would do manually, behind the scenes.
This is vastly different in dynamic languages, as in C creating the 2D array in one go, would always be the preferred way, though when creating a jagged array that is not an option.
But of course the interpreter could in theory, after parsing the code, create a 2D array in one go when you only use it like that, but I would highly doubt that, as there is not too much to gain from this rather complex optimization.

2. I think in AHK you'll always have jagged arrays, that is with possibly different lenghts in higher dimensions, there is no real CreateMatrix like command.
So, you should always be able to put other values than an array at any index, though this will make your code harder to understand.
But I would encourage you, to just try it out.

3. When you do array.InsertAt(index, [val1, val2, val3,...]) you actually insert an array, not the values of that array.
So it's technically the exact same as

Code: Select all

child := [val1, val2, val3]
array.InsertAt(index, child)

And lastly, Arrays are always objects, there is no difference in a "temporary Array", other than the interpreter will free its memory shortly thereafter, when you don't store a reference to it.

[boiler]

…in AHK this is even a bit more "inconsistent" compared to static languages, since when passing arrays to a function, by default they are indeed copied.
When you want to change the original array inside of a function, you have to declare the parameter as ByRef.
This is very different from a static language like C or Java, and an important consideration when you work with large arrays.

This is not correct. When you pass an array (or any object) as a parameter to a function in AHK, it passes a reference to the array/object. It does not create a duplicate of the array for the function to use. Here is a demo:

Code: Select all

MyArray := ["a", "b", "c"]
ChangeElement(MyArray)
Msgbox, % MyArray.1 ; displays 'hello'
return

ChangeElement(arr) {
	arr.1 := "hello"
}

Per the documentation:

Scripts interact with an object only indirectly, through a reference to the object. When you create an object, the object is created at some location you don't control, and you're given a reference. Passing this reference to a function or storing it in a variable or another object creates a new reference to the same object.

[Hardcoder]

…in AHK this is even a bit more "inconsistent" compared to static languages, since when passing arrays to a function, by default they are indeed copied.
When you want to change the original array inside of a function, you have to declare the parameter as ByRef.
This is very different from a static language like C or Java, and an important consideration when you work with large arrays.

This is not correct. When you pass an array (or any object) as a parameter to a function in AHK, it passes a reference to the array/object. It does not create a duplicate of the array for the function to use. Here is a demo:

Code: Select all

MyArray := ["a", "b", "c"]
ChangeElement(MyArray)
Msgbox, % MyArray.1 ; displays 'hello'
return

ChangeElement(arr) {
	arr.1 := "hello"
}

Per the documentation:

Scripts interact with an object only indirectly, through a reference to the object. When you create an object, the object is created at some location you don't control, and you're given a reference. Passing this reference to a function or storing it in a variable or another object creates a new reference to the same object.

Indeed, though I remember reading something in the documentation, that made me wonder why it is like that.
Of course I can't find it now
But thanks for clearing that up, I will definitely search the docs more to find out, why I misread and believed that without just testing in 1min.

[ahketype]

@Hardcoder, thanks again, that summary fits with what I discovered, except @boiler's correction. I think this function behaviour with parameters is the "same difference" as with variables pushed to a linear array (i.e. they are static) compared with pushing arrays to another array, which push just a reference. (EDIT> That is to say, for a simple variable, you have to use ByRef to avoid it being a copy that's acted on...or the "global" options.)

My little test script demos some of your other ideas - you can push either (static variable or array) to an array, the parent can be "jagged", and what I called the "rows" can be tested to see if they are objects, and hence their length or maxIndex taken.

I'm coming to this from several 'static' languages, and not used to objects even after years using them! In another language I use, I can interrogate an array, e.g. PRINT arr and it will do much as @Chunjee's print() function does, with nested brackets and commas. In AHK, accessing array will give null result if that index holds an object reference.

Code: Select all

/*
Incidentally, print() fails to handle the data and the script hangs, apparently when non-numeric values are pushed.
*/

parent := ["notanobject"]
parent.push([1, 2, 3])
gosub, Output
; print(parent)

v1 := 4, v2 := 5, v3 := 6
parent.push([v1, v2, v3])
gosub, Output
; print(parent)

parent.push([7, 8, 9, 10])
gosub, Output
; print(parent)

parent.push(["a", "b", "c"])
gosub, Output
; print(parent)

parent.removeAt(1)
gosub, Output
; print(parent)

parent.insertAt(1, ["first", "second"])
gosub, Output
; print(parent)

Exit

Output:
out := ""
Loop, % parent.MaxIndex()
{
	i := A_Index
	if IsObject(parent[i])
	{
		child := parent[i]
		out .= &child . " : "
		Loop, % child.MaxIndex()
		{
			val := parent[i, A_Index]
			if !val
				val := "-"
			out .= val . A_Tab
		}
	}
	else
	{
		out .= parent[i]
	}
	out .= "`n"
}
MsgBox % out
return

Print(obj, quote:=False, end:="`n")
{
	static cout:=FileOpen("*", "w")
	, escapes := [["``", "``" "``"], ["""", """"""], ["`b", "``b"]
	, ["`f", "``f"], ["`r", "``r"], ["`n", "``n"], ["`t", "``t"]]
	if IsObject(obj)
	{
		for k in obj
			is_array := k == A_Index
		until !is_array
		cout.Write(is_array ? "[" : "{")
		for k, v in obj
		{
			cout.Write(A_Index > 1 ? ", " : "")
			is_array ? _ : Print(k, 1, "") cout.Write(": ")
			Print(v, 1, "")
		}
		return cout.Write(( is_array ? "]" : "}") end), end ? cout.__Handle : _
	}
	if (!quote || ObjGetCapacity([obj], 1) == "")
		return cout.Write(obj . end), end ? cout.__Handle : _
	for k, v in escapes
		obj := StrReplace(obj, v[1], v[2])
	while RegExMatch(obj, "O)[^\\x20-\\x7e]", m)
		obj := StrReplace(obj, m[0], Format("\\u{:04X}", Ord(m[0])))
	return cout.Write("""" obj """" . end), end ? cout.__Handle : _
}

[Chunjee]

Code: Select all

; Incidentally, print() fails to handle the data and the script hangs, apparently when non-numeric values are pushed.

Sorry! It looks like the gist I linked is not the most up-to-date or something. This is what we use on Discord and does not have that issue:

Code: Select all

#Persistent
Print(p*) {
    static _ := DllCall("AllocConsole"), cout := FileOpen("CONOUT$", "w")
    for k, v in p
        out .= "`t" (IsObject(v) ? Print_stringify(v) : v)
    cout.Write(SubStr(out, 2) "`n")
    cout.__Handle ; Flush write buffer
}
Print_stringify(obj, quote:=False, end:="") {
    static escapes := [["``", "``" "``"], ["""", """"""], ["`b", "``b"]
    , ["`f", "``f"], ["`r", "``r"], ["`n", "``n"], ["`t", "``t"]]
    if IsObject(obj) {
        for k in obj
            is_array := k == A_Index
        until !is_array
        out .= is_array ? "[" : "{"
        for k, v in obj {
            out .= (A_Index > 1 ? ", " : "")
            . (is_array ? _ : %A_ThisFunc%(k, 1, "") ": ")
            . %A_ThisFunc%(v, 1, "")
        }
        return out . (is_array ? "]" : "}") end
    }
    if (!quote || ObjGetCapacity([obj], 1) == "")
        return out . obj . end
    for k, v in escapes
        obj := StrReplace(obj, v[1], v[2])
    while RegExMatch(obj, "O)[^\x20-\x7e]", m)
        obj := StrReplace(obj, m[0], Format(""" Chr({:04d}) """, Ord(m[0])))
    return out """" obj """" end
}

If you are getting to enjoy arrays may I suggest https://chunjee.github.io/array.ahk/
AutoHotkey lacks built-in iteration helper methods (as of 1.1.33) to perform many of the common array behaviors found in other languages. This package ports some common Array object methods to AutoHotkey's Array object.

[ahketype]

Now I'm really confused. Why aren't copies of the same address for 'letters' added to 'parent' (i.e. why don't older rows added that way change), when I didn't do letters := []?

Code: Select all

; N.B. The parameter of Output() describes the procedure about to be undertaken
; so the messagebox gives indication of the next action.

MsgBox, We start with the array [[99, 98, 97]]

parent := [[99, 98, 97]]
Output("push([1, 2, 3])")

parent.push([1, 2, 3])
Output("push([v1, v2, v3]) set as 4, 5 and 6")

v1 := 4, v2 := 5, v3 := 6
parent.push([v1, v2, v3])
Output("push(78910), a literal number")

parent.push(78910)
Output("push(letters), defined as the array [a,b,c,d]")

letters := ["a", "b", "c", "d"]
parent.push(letters)
Output("removeAt(1), remove the first row")

parent.removeAt(1)
Output("redefine variable 'letters' as [first,second]`nand then insertAt(1, letters)`nbut [a,b,c,d] won't change...")

letters := ["first", "second"]
parent.insertAt(1, letters)
Output("push(letters) this time as [12,13,14],`nWhat happens to earlier versions?...")

letters := [12, 13, 14]
parent.push(letters)
Output("push(letters) this time as [9, 8, 7],`nWhat happens to earlier versions?...")

letters := [9, 8, 7]
parent.push(letters)
Output("exit, confused. `;`)")

Exit

Output(next) {
	global
	out := "Index`tAddress`tContents`n`n"
	Loop, % parent.MaxIndex()
	{
		i := A_Index
		out .= i . A_Tab
		child := parent[i]
		if IsObject(child)
		{
			out .= &child . A_Tab
			Loop, % child.MaxIndex()
			{
				val := parent[i, A_Index]
				if !val
					val := "-"
				out .= val . A_Tab
			}
		}
		else
		{
			out .= &child . A_Tab . parent[i]
		}
		out .= "`n"
	}
	MsgBox % out .= "`n`nNow we will " . next
	return 0
}

[Hardcoder]

Now I'm really confused. Why aren't copies of the same address for 'letters' added to 'parent' (i.e. why don't older rows added that way change), when I didn't do letters := []?

Are you referring to your code?
If so, I don't see it.
Please give a concrete example.

[ahketype]

Now I'm really confused. Why aren't copies of the same address for 'letters' added to 'parent' (i.e. why don't older rows added that way change), when I didn't do letters := []?

Are you referring to your code?

Yes.

If so, I don't see it.
Please give a concrete example.

The code in my last post is a concrete example. You explained, regarding my first example, that the reason why previously added rows in the parent array changed their values was because (if I understood right) the parent array only holds an address pointer to the contents of child, so when I changed the values in child, those pointers looked up the same values.

My latest code was just an experiment to put a few other methods together of adding and removing arrays (and static variables) into the parent. Mostly, these don't have any name, they are given literally on the fly, and obviously therefore can't be changed by that method. However, I did add the named, defined, array 'letters'. Its values enter the parent array. Then I change the values in letters and add it again (not that pushing or inserting it should change the logic - it's changed the contents of the array called 'letters'), so I expected to see a difference - just as in the first example - where the objects already inserted into parent would be address pointers to 'letters' and thus be replaced in the messagebox by the new values. They aren't. They stay as they were entered. I don't understand why.

EDIT > The 'fix', if you remember, was re-using the syntax, child := [], and I do not use letters := [] in the last example. But each use seems to create a new memory location, a new version of the array, with its address left as it was in the row of parent as entered.

I can only guess there's something different about the way the whole is constructed here that changes the behaviour. See?

[ahketype]

I'm sorry it's a rather complicated example. I'll try to write something more concise, and I might figure out the answer to my confusion in the process. In the morning, that is.

[Chunjee]

I did add the named, defined, array 'letters'. Its values enter the parent array. Then I change the values in letters and add it again (not that pushing or inserting it should change the logic - it's changed the contents of the array called 'letters'), so I expected to see a difference - just as in the first example - where the objects already inserted into parent would be address pointers to 'letters' and thus be replaced in the messagebox by the new values. They aren't. They stay as they were entered. I don't understand why.

You haven't created any pointers in your example

Code: Select all

; We start with the array [[99, 98, 97]]
parent := [[99, 98, 97]]
; create letter var
letter := "a"
; create pointer to letter var
pointer := &letter
parent.push(pointer)
; change letter var
letter := "b"
; parent is now: [[99, 98, 97], 53901952]
msgbox, % strGet(parent[2]))
; => "b"

I am not familiar enough with pointers to an object; so I did a single variable instead