IsByRef

[lexikos]

You can't.

That's poor. Why didn't you take over IsByRef() (or a replacement) from v1.1?

Reason #1:

IsByRef has been removed due to ambiguity in the implementation (an alias may be due to passing a reference in, taking a reference to the parameter itself, or referring to it in a closure).
Source: Preview of changes: scope, function and variable references - AutoHotkey Community

AutoHotkey's functions each have exactly one set of local variables (henceforth "Vars" to show that we're talking about an implementation detail, not semantics from an external perspective):

Each reference to a variable or function is resolved to a memory address, unless it is dynamic.
Source: Script Performance | AutoHotkey v2

Putting aside closures for the moment, if a function already has a running "instance" (whether it is recursive or was interrupted by another thread), calling it causes all of its local Vars to be "backed up" and unset, ready for the new instance. When that instance returns or exits, the Vars are restored to their former state (including any aliasing). One likely reason for this approach is that early versions of AutoHotkey had only global variables and no functions. In any case, this is how local variables work.

In v1, this is a problem for recursive functions. When a function passes its own Var to itself ByRef, the new call to the function receives a reference to its own (now empty) local Var.

If a closure or &ref actually aliased a local Var, it would cease to have the correct value when the outer function returns or is called recursively or by an interrupting thread. So it doesn't work that way. Instead, any local Var which needs to outlive its function automatically becomes an alias for another Var.

For closures, the outer function allocates a block of FreeVars (free variables; "free" as in "not constrained by the duration of the function call") and these are aliased by both the outer function's "down vars" (local Vars which get captured) and by each closure's "up vars" (Vars which just act as pointers, local to the closure). This is the means by which the outer x and the inner x in outer(x) => inner() => x refer to the same storage space despite the functions potentially running at different times.

For &X, when a VarRef is needed for the first time, one is constructed with X's former value and X becomes an alias for the VarRef. If X is local, this lasts until the function returns.

In (x) => () => &x, there are four Vars:

1. When the outer function prepares to execute, it allocates a reference-counted block of FreeVars, containing one Var.
2. The first x is the "down var", an alias for the FreeVar of the currently running instance of the outer function.
3. The second x is the "up var", an alias for the FreeVar of the currently running instance of the closure.
4. When &x is evaluated, a VarRef is constructed and the value is moved from the FreeVar into the VarRef. Both the inner x and the FreeVar then become aliases for the VarRef. The outer x is unchanged, and may or may not even be aliasing the same FreeVar still.

In (&x) => () => &x, the outer x is "an alias for the caller's variable". But because it needs to be an alias for the FreeVar in order for the closure to work, really the FreeVar is an alias for the caller's VarRef. When the closure prepares to execute, a pointer to the caller's VarRef is copied from the FreeVar into the inner x (removing one level if indirection and making it a direct alias, for performance), and &x returns the caller's VarRef. It is effectively the same as (x) => () => x, except that only a VarRef can be passed in.

For a ByRef parameter specifically, it can be:

• A normal local Var, not an alias.
• An optimized alias for the caller's Var.
• An alias for the caller's VarRef.
• An alias for a locally constructed VarRef.
• An alias for a FreeVar, which is a normal local Var.
• An alias for a FreeVar, which is an alias for a VarRef constructed by a closure.

If it is an alias for a FreeVar, that could be either one that the current instance allocated or one that the caller (which could be the same function) passed.

For error-reporting purposes, a VarRef retains the name and "scope flags" of the source variable. It is possible to determine, for instance, that the parameter is an alias for a VarRef sourced from a parameter with the same name. However, it could be a parameter of a different function, a parameter of a different call to the same function, or the same parameter of the same call, having used &ref.


Reason #2:

It is redundant.

If you want to know whether a value was provided for a parameter, just declare it as p:=unset or p? and use IsSet(p). If you want to pass the parameter along, omitting it if unset, just use F(p?). If you want to be verbose, you can use F(IsSet(p) ? p : unset).

If you want to do the same with a parameter that accepts a VarRef - that is, if you want to know what value was passed for the parameter - then just declare the parameter as by-value. Why complicate it by making the parameter optionally an alias and then afterward asking whether a value was passed?

A parameter doesn't need to be marked with & when it is intended to take a variable reference. MP(X?, Y?) => MouseGetPos(X?, Y?) will pass along whatever VarRef you give MP, or omit the parameter(s). Why would you want to use MP(&X?, &Y?) => MouseGetPos(IsByRef(X) ? &X : unset, IsByRef(Y) ? &Y : unset)?


Reason #3:

Semantics and consistency within the language.

You would presumably want to call IsByRef(X) or IsByRef(&X).

As a function, IsByRef(X) should throw an error if X is unset, or an alias to an unset variable. In theory, evaluating X should produce the value of the target variable, not a reference to the alias. Permitting this call would mean making IsByRef an exception like IsSet; an operator rather than a function. This would mean another reserved word, and that it couldn't be called by reference, only by directly using the keyword. The exception makes sense for IsSet because its express purpose is to check whether the variable is set, and it also needs to affect load-time warnings. It makes less sense for IsByRef, even putting aside backward-compatibility.

IsByRef(&X) should create a VarRef and pass it to IsByRef. A VarRef is "a value representing a reference to a variable". Nowhere is it stated that a VarRef can be used to identify the original variable, or that it is an alias for the original variable. Without IsByRef, the distinction doesn't matter. It is intentionally left vague because in the current implementation, in ((&X) => X)(&Y), both X and Y become aliases for the VarRef. Even putting aside the implementation, X and Y could be interpreted as equivalent references to a variable or storage location.


Reason #4:

Avoiding the imposition of constraints on future implementations and semantics.

There is probably one way to resolve the ambiguity between a parameter which is an alias of the caller's variable and a parameter which is an alias for other purposes. That is to store additional information which has no other purpose than to implement IsByRef; specifically, making a distinction between aliases created for the purpose of ByRef and other aliases.

Making this distinction now means imposing additional requirements on all backward-compatible revisions to the implementation, or alternative implementations that might be created in future.


Reason #5:

Adhering to other lines of thought.

In Concepts - Variables, I explained variables in terms that I believe consistent with how Chris presented them; where a variable has one name, one scope, one storage location. There's another common line of thought that I think is more elegant ("pleasingly ingenious and simple"), and is directly taught with certain other languages. In that line of thought, one or more names can be bound to a value/object/location. A variable doesn't have to inherently be something that has its own name.

As with IsByRef imposing constraints on the implementation (#4), it would impose constraints on how one can think of a variable when explaining what IsByRef does. Different people think different ways, so supporting different ways of thinking is helpful.


This still isn't the full extent of my thoughts, but this post has already gone way "over-budget", so I will leave it at that.

[just me]

Hi @lexikos,
thanks for the very detailed explanation.

My only reason for asking for IsByRef() is performance respectively the possibility to skip unnecessary code execution.
This method (https://www.autohotkey.com/boards/viewtopic.php?f=83&t=95389)

Code: Select all

      ; ----------------------------------------------------------------------------------------------------------------
      ; METHOD Step        Execute the statement and get next row of the query result if available.
      ; Parameters:        ByRef Row   - Optional: Variable to store the row array
      ; Return values:     On success  - True, Row contains the row array
      ;                    On failure  - False, ErrorMsg / ErrorCode contain additional information
      ;                                  -1 for EOR (end of records)
      ; ----------------------------------------------------------------------------------------------------------------
      Step(&Row?) {
         Static SQLITE_INTEGER := 1, SQLITE_FLOAT := 2, SQLITE_BLOB := 4, SQLITE_NULL := 5
         Static EOR := -1
         Local Blob, BlobPtr, BlobSize, Column, ColumnType, RC, Value
         This.ErrorMsg := ""
         This.ErrorCode := 0
         If !(This._Handle)
            Return This._SetError(0, "Invalid query handle!")
         RC := DllCall("SQlite3.dll\sqlite3_step", "Ptr", This._Handle, "Cdecl Int")
         If (RC = This._DB._ReturnCode("SQLITE_DONE"))
            Return (This._SetError(RC, "EOR") | EOR)
         If (RC != This._DB._ReturnCode("SQLITE_ROW"))
            Return This._SetError(RC)
         This.CurrentStep += 1
         If !IsSetRef(&Row)
            Return True
         Row := []
         RC := DllCall("SQlite3.dll\sqlite3_data_count", "Ptr", This._Handle, "Cdecl Int")
         If (RC < 1)
            Return True
         Row.Length := RC
         Loop RC {
            Column := A_Index - 1
            ColumnType := DllCall("SQlite3.dll\sqlite3_column_type", "Ptr", This._Handle, "Int", Column, "Cdecl Int")
            Switch ColumnType {
               Case SQLITE_BLOB:
                  BlobPtr := DllCall("SQlite3.dll\sqlite3_column_blob", "Ptr", This._Handle, "Int", Column, "Cdecl UPtr")
                  BlobSize := DllCall("SQlite3.dll\sqlite3_column_bytes", "Ptr", This._Handle, "Int", Column, "Cdecl Int")
                  If (BlobPtr = 0) || (BlobSize = 0)
                     Row[A_Index] := ""
                  Else {
                     Blob := Buffer(BlobSize)
                     DllCall("Kernel32.dll\RtlMoveMemory", "Ptr", Blob, "Ptr", BlobPtr, "Ptr", BlobSize)
                     Row[A_Index] := Blob
                  }
               Case SQLITE_INTEGER:
                  Value := DllCall("SQlite3.dll\sqlite3_column_int64", "Ptr", This._Handle, "Int", Column, "Cdecl Int64")
                  Row[A_Index] := Value
               Case SQLITE_FLOAT:
                  Value := DllCall("SQlite3.dll\sqlite3_column_double", "Ptr", This._Handle, "Int", Column, "Cdecl Double")
                  Row[A_Index] := Value
               Case SQLITE_NULL:
                  Row[A_Index] := ""
               Default:
                  Value := DllCall("SQlite3.dll\sqlite3_column_text", "Ptr", This._Handle, "Int", Column, "Cdecl UPtr")
                  Row[A_Index] := StrGet(Value, "UTF-8")
            }
         }
         Return True
      }

should execute the code to fill the Row parameter only if a VarRef has actually been passed. Otherwise it's a senseless waste of resources. Also, like all built-in functions it should work with VarRefs to uninitialized variables.

As I understand your answers, there is no way and there will be no to achieve this!?!

BTW: Next() is just an alias for Step(). That's why I thought the parameter definitions should be identical.

[lexikos]

My only reason for asking for IsByRef() is performance respectively the possibility to skip unnecessary code execution.

If the parameter isn't ByRef, you can do that! That's why I kept saying "just remove &"!

There is nothing a ByRef parameter can do that a normal parameter can't do, in functional terms. Consider a function:

Code: Select all

X(&Y) {
    if !IsSet(Y)
        Y := 42
}

This is equivalent:

Code: Select all

X(Y) {
    if !(Y is VarRef)
        throw TypeError("Parameter #1 of X requires a variable reference, but received a" (Type(Y) ~= 'i)^[aeiou]' ? "n " : " ") Type(Y) ".", -1, Y)
    if !IsSet(%Y%)
        %Y% := 42
}

It's easy to see that if you use the second form, you can check for yourself whether a VarRef was passed.


If you have a section of code which performs some computation that you want to avoid when there's no output variable, that can (and maybe should) be its own function. You can take the VarRef as a value and pass it to the other function, or skip that if the parameter is unset. Then you have a function where the parameter is either a VarRef or unset, and a function where the parameter is always an alias for the caller's variable. Both parameters have zero ambiguity.

Code: Select all

F(ref) {
    IsSet(ref) && compute(ref)
    compute(&alias) {
        IsSet(alias) || alias := 0
        alias++  ; Multiple references, no need for explicit %dereferencing%.
        alias++
        alias++
    }
}
F(&G)  ; G is unset
MsgBox G
F(&G)  ; G is not unset
MsgBox G

v2.1-alpha.4 supports (anonymous or named) function definitions inside expressions.

Code: Select all

F(ref) {
    IsSet(ref) && ((&alias) {
        alias ??= 0
        alias++  ; Multiple references, no need for explicit %dereferencing%.
        alias++
        alias++
    })(ref)
}
F(&G)  ; G is unset
MsgBox G
F(&G)  ; G is not unset
MsgBox G

[just me]

So this is what is required to solve my special problem in Class SQLiteDB using basic v2.0 syntax:

Code: Select all

; ==================================================================================================
; Alias        -  can be used as an alias for Original()
; Parameters:
;     Var      -  optional: if not omitted it must be a VarRef to retrieve an additional result
; ==================================================================================================
Alias(Var?) { ; !!!!! Note: If Var is not omitted is must be a VarRef !!!!!
   Switch {
      Case !IsSet(Var):
         Return Original()
      Case (Var Is VarRef):
         Return Original(Var)
   }
   Throw TypeError("Parameter #1 requires a variable reference, but received a" .
                   (Type(Var) ~= 'i)^[aeiou]' ? "n " : " ") . Type(Var) ".", -1, Var)
}
; ==================================================================================================
; Original     -  retrieves some values
; Parameters:
;     Var      -  optional: if not omitted it must be a VarRef to retrieve an additional result
; ==================================================================================================
Original(Var?) { ; !!!!! Note: If Var is not omitted is must be a VarRef !!!!!
   If IsSet(Var) && !(Var Is VarRef)
      Throw TypeError("Parameter #1 requires a variable reference, but received a" .
                      (Type(Var) ~= 'i)^[aeiou]' ? "n " : " ") . Type(Var) ".", -1, Var)
   ; ...
   ; ... do some always required stuff
   ; ...
   ; Now check a second time, whether a VarRef has been passed
   If IsSet(Var) {
      Result := 0
      Result++
      Result++
      Result++
      %Var% := Result
   }
}

Somehow not satisfactory but apparently working.

[lexikos]

Why would you require this type-checking for VarRef parameters, when you don't type-check other parameters?

If you want robust parameter validation, the code required for optional VarRef parameters is entirely comparable to that of other types. If you don't want to bother with parameter validation, then don't, and the usage of the VarRef parameter will be comparably trivial to that of other types. If you're going to be pedantic or lazy, at least be consistent.

Something it seems I've failed to mention: It was always my plan to extend &ref parameters to support more types of objects. Partly to support f(&x.y), but also to support ComValueRef or user-defined types. So it may be true in current versions that a ByRef parameter requires a VarRef, but a future version may permit any object, or any object with specific properties (which I haven't decided on, but could be []; i.e. __Item with no parameters, since ComValueRef already implements it).

The current version requires a VarRef because of implementation details. In v1, the parameter would be converted to a non-alias if you passed something other than a variable reference. One reason (maybe the main reason?) I didn't permit this in v2 was to ensure that scripts won't come to rely on other objects being passed by value to a ByRef parameter.

I initially implemented __Item in VarRef but removed it at the last minute, because I'm not sure about overloading that property in particular. If I hadn't removed it, you could cleanly permit other objects to stand in for a VarRef (instead of a bit uncleanly, using both %Var% and Var[], or something else). For example:

Code: Select all

(Any.DefineProp)(VarRef.Prototype, '__Item', {
    Set: (&ref, value) => ref := value,
    Get: (&ref)        => ref         
})

F(V?) {
    if IsSet(V)
        V[] := InputBox(,,,"V was provided").Value
    else
        MsgBox("V was not provided")
}

F(&G)
MsgBox G
F()
vbuf := Buffer(24, 0)
vref := ComValue(0x400C, vbuf.ptr)
F(vref)
MsgBox vref[]

Another possibility for dealing with optional parameters in v2.1-alpha.2+ is optional chaining.

Code: Select all

#Requires AutoHotkey v2.1-alpha.2+

(Any.DefineProp)(VarRef.Prototype, 'Set', {Call: (&ref, value) => ref := value})
(Any.DefineProp)(VarRef.Prototype, 'Get', {Call: (&ref)        => ref         })

F(V?) {
    V?.Set(InputBox(,,,"V was provided").Value)
        ?? MsgBox("V was not provided")
}
F(&G)
MsgBox G
F()

V?.[] or V?.__Item is the equivalent for __Item, but assigning to an optional chain (V?.[] :=) is currently not permitted. I chose to copy JavaScript in not allowing it for now, but may change it if I don't find a good explanation for that restriction.

[just me]

Thanks, @lexikos. I think this has already taken up too much of your time.

[lexikos]

When defining a function, one or more of its parameters can be marked as optional.
Append := followed by a literal number, quoted/literal string such as "fox" or "", or an expression that should be evaluated each time the parameter needs to be initialized with its default value. For example, X:=[] would create a new Array each time.
...
ByRef parameters also support default values; for example: MyFunc(&p1 := ""). Whenever the caller omits such a parameter, the function creates a local variable to contain the default value; in other words, the function behaves as though the symbol "&" is absent.
Source: Functions - Definition & Usage | AutoHotkey v2

If you combine these two points, there should be at least two ways to allow the remainder of the function to know that the parameter was omitted:

1. Let the default value be a unique sentinel that can't exist in any caller's variable.
2. Give the parameter's default expression a side-effect.

Code: Select all

#Requires AutoHotkey v2.0.8

a(&r := defaulted() => 0) {
	MsgBox "a: r was " ((r ?? 0) = defaulted ? "omitted" : "specified")
}

b(&r := (default_r := true, unset)) {
	MsgBox "b: r was " (isSet(default_r) ? "omitted" : "specified")
}

a()
a(&g)
b()
b(&g)

This only works correctly in v2.0.8+, because previous versions mistakenly relied on ?? internally to determine whether to assign the default value.

I still think it's cleaner to avoid ByRef.