Getting to first/next key in custom enum/next() Topic is solved

[sirksel]

After re-reading a very helpful post from Lexikos on custom collection objects, I'm trying to change around some of my objects to support custom enumerators. However... I'm having trouble figuring out how to get at the first/next non-numerical key when you're overriding the _newenum() and next(). I've seen lots of examples of enumerating only integer keys, and I've got that implemented. You start at one and go from there.

For non-integer keys (or more precisely, the superset of all keys, which may include integers), the only way I could think of to get a custom enumerator to work is to enumerate all keys to a list in a for-loop in the __new() of the object returned by _newenum(). Then, in next(), just navigate through the list of keys and call the next one. For large dictionaries, this doesn't seem like a good practice... enumerating all keys just to get to the first. I'm sure there's a better way, and I'm hoping someone can point me in the right direction.

I thought about maybe having a shadow enumerator (of the standard type) created in the __new() of the enumerator class, by calling .base._newenum(). Then maybe in my custom next(), I could just call that shadow enumerator's next and alter the output from there before I return. I wasn't able to get that to work.

I've tried to boil it down to the simplest case, below. I have a base iterable class that defines _newenum() as returning an object of enumerator class. Let's say the only transform I want to do is to change my custom next() to make the value (not the key) return if only one byref parameter is passed. That would enable the syntax for itm in collection rather than requiring for key, itm in collection. In the standard behavior, if you supply one variable, you get the key not the value. (I'm not saying that my custom enumerator is a better treatment... only that it's a simple example of something I could do to alter the standard enumerator behavior for not-necessarily-numerical keys.)

I think I can figure out my more complicated customizations if you can help me just solve this one:

Code: Select all

class itr extends obj {   ; iterable base class
	_newenum() {
		return new enm(this)
	}
}
class enm {   ; enumerator base class
	__new(itr) {   ; will work for any iterable
		this.itr := itr
	}
	next(byref var1, byref var2:="") { 
		; I want this to return val (not key) if only var1 is specified
		; or key, val if two vars are specified.
	} 
}

(I'm writing in v2, in case there's some capability there that could be helpful that's not also in v1.) Thanks in advance for your help.

[sirksel]

UPDATE: Here was my quick-and-dirty attempt that I referenced to call the standard enum before I overrode it. This code fails because base._newenum() returns an empty string. I'm definitely misunderstanding things, but I'd appreciate any pointers you all could give me.

Code: Select all

; NOTE: code below is AHK v2
class itr {   ; iterable base class
    _newenum() {
		return new enm(this, base._newenum())
	}
}
class enm {   ; enumerator base class
    __new(itr, std) {   ; will work for any iterable
        this.itr := itr      ; iterable on which the enum was called
        ,this.std := std     ; standard iterator that was overridden
	}
    next(byref var1, byref var2:='') {
        if not this.std.next(key, val),  return var1:=var2:='', 0
        return var2 is byref ? (var1:=key, var2:=val, 1) : (var1:=val, 1)
	}
}

x := [10,20,30]
x.base := itr
for itm, val in x
    msgbox % itm ':' val
exitapp

[lexikos]

Each method also has an equivalent function, which can be used to bypass any custom behaviour implemented by the object -- it is recommended that these functions only be used for that purpose. To call one, prefix the method name with "Obj" and pass the target object as the first parameter. For example:

Code: Select all

array := [1, 2, 3]
MsgBox % ObjMaxIndex(array) " = " array.MaxIndex()

Source: Object

For _NewEnum, exclude the underscore -> ObjNewEnum(object).

Making the built-in methods more consistent with user-defined methods (with respect to calling them from a derived class which overrides them) is on my list for an unspecified future version. In other words, base.Method() only works with user-defined methods, but I want it to work with all methods.

if not this.std.next(key, val), return var1:=var2:='', 0

The standard enumerator does not clear the variables. In v2-a075, it generally has no effect since the for-loop restores the variables to their pre-for-loop values when the enumerator returns false.

return var2 is byref ?

You need to pass a string to the is operator. byref is just an empty variable.

[sirksel]

Lexikos, as always, thanks so much! After your fixes, it works like a charm. An answer of yours is as good as gold, and always gives me a little more to think about each time. And, not surprisingly, it also usually involves you quoting some section of the documentation that I didn't properly read/digest... Sorry about that. I know you are busy, and yet you are generous with your time in the community. I'm very grateful that you take the time to respond personally. One day, because of you and a few others in this forum, I will actually be a decent AHK programmer!

Just curious... is your general impression that this is the right way to go about customizing a not-necessarily-integer-keyed enumerator -- with the shadow base enumerator behind it. I looked at lots of examples in the forum and boards, and several build the entire key list upfront rather than shadowing with a live base enumerator. That didn't seem efficient. I have a number of customizations I'm doing on various objects' enumerators... but they all rely on this same fundamental issue for the non-integer-keyed variety... to have a way to step through keys that don't necessarily start at 1.

Is there any better way that I'm missing?

Lexikos, thanks again for your help! For anyone else interested in the finished code, here it is:

Code: Select all

; NOTE: code below is AHK v2
class itr {   ; iterable base class
    _newenum() {
		return new enm(this, objnewenum(this))
	}
}
class enm {   ; enumerator base class
    __new(itr, std) {        ; should work for any iterable
        this.itr := itr      ; iterable on which the enum was called
        ,this.std := std     ; standard iterator that was overridden
	}
    next(byref var1, byref var2:='') {
        if not this.std.next(key, val),  return 0
        return var2 is 'byref' ? (var1:=key, var2:=val, 1) : (var1:=val, 1)
	}
}

itms := {a:10,b:20,c:30,base:itr}
for key, itm in itms
    msgbox % key ':' itm 
for itm in itms
    msgbox % itm 
exitapp

[lexikos]

That is probably the best way.

I suppose there's no need for this.itr, though.

Also, I suppose you can skip the assignment in the two-var case by using var1, var2 instead of key, val.

An experiment:

Code: Select all

; NOTE: code below is AHK v2
class itr2 {   ; iterable base class
    _NewEnum() {
		return new enm2(ObjNewEnum(this))
	}
}
class enm2 {   ; enumerator base class
    __new(std) {
        this.std := std
	}
    Next(ByRef var1, ByRef var2:='') {
        if var2 is 'ByRef'
           return this.std.Next(var1, var2)
        ; this.Next(var1) -> this.std.Next(this, var1)
        this.Next := ObjBindMethod(this.std, "Next")
        return this.Next(var1)
	}
}

itms := {a:10,b:20,c:30,base:itr2}
for key, itm in itms
    msgbox % key ':' itm 
for itm in itms
    msgbox % itm 
; OK so far.

e := itms._NewEnum()
e.Next(a)
e.Next(b, c)
MsgBox % a "," b "," c  ; No good.

It can be a little faster this way.

[sirksel]

Thanks, Lexikos... many good ideas come from your "experiments"... The only part I don't understand is why this works:

Code: Select all

	;this.Next(var1) -> this.std.Next(this, var1)
	this.Next := ObjBindMethod(this.std, "Next")
	return this.Next(var1)

I think that subsequent calls to enm2.Next are actually calling the standard iterator's .Next, but then how does it return the value instead of the key with just one variable? Does your comment above the binding explain the trick as the fact that the bound method passes this as its first param? Does that this actually get used to refer to the right object or does it just "consume" the first byref var? Does it serve both purposes or just the latter? I keep talking myself through it and then realizing I don't understand why this works... Sorry if I'm being dense.

[lexikos]

x := ObjBindMethod(y, z) produces a function such that calling %x%() or x.Call() has the effect of y[z](). y[z] isn't actually a function, so "this" isn't actually part of the parameter list at this point.

When you call this.Next(v), the function referred by this.Next receives this as its first parameter, like this.Next.Call(this, v). You are passing this to the bound method, and the bound method just passes all parameters along to the original object, making the call y[z](this, v); that is, (this.std)["Next"](this, var1).

Actually, the enumerator modifies the this variable, but since you're returning immediately it doesn't matter. On subsequent calls, this is a plain object reference passed by the for-loop, not a variable.

[sirksel]

Thanks so much. I think I get it, at least to this point:

...making the call y[z](this, v); that is, (this.std)["Next"](this, var1).

But, I guess that's where I start to lose the logic. So when I call an object's Next directly, like thisenum.Next(key, val), isn't the this implicitly the first parameter there too, but it's just hidden? In other words, we don't code it into the method (as you would self in Python), but the this is just immediately available for use because it is passed implicitly?

So how/why does the (this.std)["Next"](this, var1) behave differently? Sorry I'm not seeing this more clearly. Is this being passed, either implictly or explicitly, regardless of whether (i) I call the .Next in a regular enumerator directly, (ii) the for..in loop calls it or (iii) our bound-method object calls it? Maybe another way of asking... Is it that the way the parameters are placed in the call is different or the way they're consumed is different in these three cases? In still other words... It seems like it's the case that either one "fewer" implicit+explicit parameter is getting passed in our bound-method case or one "extra" parameter is being consumed, but I'm somehow not seeing it.

I'm sorry to have to ask a follow-up, but somehow the light bulb still hasn't clicked in my head. I'm just hopeful that your remedial advice to me helps some other poor lost soul in the forum with a similar question one day... Thanks so much for the help!

[lexikos]

It does not behave differently.

You have an object O1, with two methods, M1 and M2.

• M1 is really a function, O1.M1 := Func("F").
• M2 is really a bound method, O1.M2 := BM := ObjBindMethod(O2, "Mx").

When you call O1.M1(), the function needs to know which object is being operated on, so the object calls F(O1), or in other words, O1.M1.Call(O1).

When you call O1.M2(), O1 does the same thing: it passes itself as the first parameter of the bound method: O1.M2.Call(O1).

When you call O1.M2(p), your parameter is simply appended to the parameter list: O1.M2.Call(O1, p).

Both parameters (O1 and p) are passed to the original method as explicit parameters, and the original object (first parameter of ObjBindMethod) is passed implicitly as context.


this.Next(var) does not find some function and then pass this to it as a parameter. It merely invokes the object, passing "Next" and var as parameters.

When this is a user-defined object, the object removes the parameter "Next" and uses it to look up which function to call. The object then inserts this at the beginning of the parameter list and calls the function. You didn't have to define the parameter explicitly because of "syntax sugar" - i.e. you wrote it inside a class definition - but it's really just a normal function parameter (see Prototypes).

If this is a built-in object, the method is implemented directly, without any need for magical transformations of the parameter list. However, whether or not there is a hidden "this" parameter within the Enumerator object's implementation cannot be observed from the outside - it's only between the Enumerator object and its implementation. Only the explicit parameters act as output variables.