[v2.1] Passing unset VarRef

[Descolada]

The improved unset variable load-time validation added in v2.1-alpha.2 apparently affects the passing of VarRefs through functions. In v2.1-alpha.4 this causes a load-time error, when in v2.0 it didn't:

Code: Select all

F(&out?) {
    G(&out?)
}
G(&out2?) {
    out2 := out2 ?? "default"
}

F(&out:="test")
OutputDebug out

whereas this works

Code: Select all

F(&out?) {
    G(&out)
}
G(&out2?) {
    out2 := out2 ?? "default"
}

F(&out:="test")
OutputDebug out

Although the working one is perfectly valid, it's hard to deduce from reading G(&out) that originally out was an optional argument. G(&out?) conveys that more clearly.
I wish that propagating optional VarRefs through functions was allowed in the same manner as in v2.0.

[V2User]

G(&out?) conveys that more clearly.

Although ? is absent in the parameter definition. But ? will follow ref everywhere, every time the ref variable is used. Therefore, it has been clear enough, I think.

image.png (2.81 KiB) Viewed 2191 times

&out in G(&out) will just create a local variable named out if out is unset, just as creating any other new local variables. Nothing special. Is that right?
Creating variables is a main use of &. Someone would probably know it may be unset.

[lexikos]

@Descolada
I think you are under a mistaken impression about what &out? does, did, and/or can possibly do.

Previously what it did was mark out as potentially unset, which had no actual effect because &out already permits out to be unset, and always returns a VarRef.

out is unset not only when you omit the parameter, but also when you pass an unset VarRef, because out is ambiguously either a local variable or an alias of the caller's variable. This ambiguity is at the core of what an optional ByRef parameter is.

The error is correct. As per the operator precedence table, the ? operator has higher precedence than &. This means ? is being applied to the variable, which is being passed to &, not the parameter. & can't accept unset. This wasn't detected in v2.0 because the operator was implemented in a very simple way that actually doesn't obey operator precedence rules, or even evaluate as an operator (it just modifies the variable reference).

When you call This.Step(&Row) (and This.Step(&Row?) in v2.0), you are always supplying a value for the parameter. If Row was omitted, you are passing a reference to the local Row variable. If it was not omitted, &Row returns the original VarRef which was passed to Next.

The valid contexts for unset are detailed in the v2.1-alpha.2 documentation, under "An unset expression may consist of" and "Unset expressions are valid only within the following contexts".

If what you want is for this.Next(&Row) to result in this.Step(&Row) and this.Next() to result in this.Step(), what you need to do is remove & and use only ?. When you declare the parameter as ByRef, unset does not reliably indicate whether the parameter was omitted: Row could be an alias to an unset variable, or it could be just an unset variable. So even if &Row? was to evaluate as something like IsSet(Row) ? &Row : unset, it probably isn't what you want.

See also IsByRef - AutoHotkey Development.

[Descolada]

@lexikos, I was under the impression that the maybe operator is mainly necessary to
a) Allow optional arguments in functions, such as F(arg?) {...} which can be called as F() and F(value)
b) Prevent accessing an unset variable, as in the case of calling F(arg?) {...} with F(value) where value was previously unset.

This means that in the case of propagating arguments through functions, normal ByVal arguments need the maybe operator for both argument definition and function call (to prevent accessing unset variable):

Code: Select all

F(out?) {
    G(out?)
}
G(out?){
}

In this case G(out?) not only prevents accessing an unset variable, but its existence signals that out *can* be unset, and also that it's most likely being propagated to the next function unchanged. For example, reading a function declaration such as this

Code: Select all

F(out?) {
    ; insert 50 lines of code here
    G(out)
}

I immediately know that in the 50 lines of codes out has very likely been set, or there is a bug in the code.

In the case of ByRef, only case a) applies because a VarRef is always an object, so naturally this will be valid:

Code: Select all

F(&out?) {
    ; insert 50 lines of code
    G(&out)
}
G(&out2?) {
    if IsSet(out2) {
        Sleep 10000
        out2 := "result"
    } else
        out2 := "default"
}

In this case it's rather important to know whether out has been set in the 50 lines of code, because 10 seconds of runtime speed depends on it. If the totally unnecessary (at least in this case) maybe operator was permitted then the use of G(&out?) instead of G(&out) would likely mean that out can be unset and is most probably being passed on unchanged.

Sure, comments and syntax highlighting can substitute that well. I simply don't understand why creating a breaking change is necessary in a case where the existence of ? can't foreseeably (at least to me) create a bug nor create ambiguities, and in some cases can be helpful.

Even the error message for using F(&out?) or G(&out?) is confusing:

This operator's right operand must not be unset.

The right operand to & can certainly be unset, just can't be followed by the maybe operator nor be explicitly unset.
This is fine:

Code: Select all

out:=unset
F(&out)

This is not:

Code: Select all

F(&(out:=unset))

Yet they should be the same...

Fair warning though: I am biased towards disliking this change, because I had to fix two libraries of mine to be compatible with v2.1 solely because of this change...

[iseahound]

@Descolada It's meant to be modeled on the Maybe monad (in some sense). Also, it wouldn't be a breaking change since functionality wasn't changed, only clarified to be less ambiguous.

[lexikos]

... would likely mean that out can be unset and is most probably being passed on unchanged.

As Row? is equivalent to IsSet(Row) ? Row : unset, &Row? would be equivalent to IsSet(Row) ? &Row : unset (or the invalid &(IsSet(Row) ? Row : unset)).

What it means is that the parameter will be omitted in the second call if it was not omitted but was an unset VarRef, or if it was omitted, in the first call. If all you do is use this to "forward" the parameter on, the second function has different behaviour to the first.

Code: Select all

A(&x?) {
    B(IsSet(x) ? &x : unset)
}
B(&y:="default") {
    MsgBox y
}
A(&g) ; y gets it default value!
A(&g := "not default")

I simply don't understand why creating a breaking change is necessary ...

Fixing a bug is not a "breaking change". What you don't understand is apparently the explanation I already gave in my previous post, or the longer one in the IsByRef topic.

The right operand to & can certainly be unset

In &x, the operand is x itself, not its value (if any). &unset isn't valid. &(1 ? x : unset) isn't valid. &(f()?) isn't valid.

Otherwise, as I've said before, the fact that x can already be unset is all the more reason why the ? suffix shouldn't be allowed. If you know that &x permits x to be unset, why are you wanting a suffix to tell you that it can be unset? What if it does nothing, like before? It doesn't mean anything. You were just misleading yourself by using it.

[iseahound]

If you can do:

Code: Select all

a := IsSet(Row) ? &Row : unset
fn(a?)

why not just allow that in a single line? (disregarding any misconceptions of what &row? actually means.

Code: Select all

fn(&row?)

[lexikos]

disregarding any misconceptions

No. Why would I do that? Make it convenient to write code that will mislead users and fail to work as intended?

[iseahound]

oh i see. In the parameters, it could be read as &(out?) while the exact same notation in the function body could be read as (&out)?

This seems like a user friendly choice

[sirksel]

I've gone back over this several times, and I'm still confused. My test code is similar to the OP:

Code: Select all

f(x, &y?) => (y:=x*3, x*2)
g(x, &y?) => (x:=2, f(x, &y?))
MsgBox g(4)

I want g() to be flexible enough to accept 1 or 2 parameters. I think I understand lexikos' statement that:

&Row? would be equivalent to IsSet(Row) ? &Row : unset

I think also understand the ambiguity as to the receiving function (paraphrasing) looking at a missing varref or a passed varref that was never set in the caller.

So... @Descolada, since it looks like we were trying to do something similar in our respective libraries... did you decide that this was the correct/best approach (eliminating the ? on the propagating call):

Code: Select all

f(x, &y?) => (y:=x*3, x*2)
g(x, &y?) => (x:=2, f(x, &y))  ;change &y? to &y

Or did you find a better approach as you were modifying your libraries? Many thanks.

[Descolada]

@sirksel I don't know what the "best" approach is.

The crux of the issue is determining whether an optional argument was provided a variable or not, and whether that VarRef variable was unset or not. Meaning in the context of function f f(&x?), x will be unset both in the case of x := unset, f(&x) and f() and the two situations can't be differentiated.

If efficiency isn't too important then I decided to ignore the case of an argument not being provided at all, and handled that as if an unset VarRef was passed. Meaning

Code: Select all

f(x, &y?) => (y:=x*3, x*2)
g(x, &y?) => (x:=2, f(x, &y))
g(2)

would be slightly inefficient as some unneeded calculations are done, but the lost efficiency is very tiny.

If efficiency is required then I opt for ByVal arguments instead, at the cost of slightly higher risk of bugs and more validation steps:

Code: Select all

f(x, y?) => ((IsSet(y) && y is VarRef && %y%:=x*3), x*2)
g(x, y?) => (x:=2, f(x, y?))
g(1) ; calculation not done
g(1, &x) ; calculation done and x is set to 6

[sirksel]

This makes complete sense now. Thank you @Descolada!

I think what you outlined seems like the best solution at present. Somehow, it would be nice if there were better syntax for either (1) streamlining this assignment syntax with an operator or (2) creating a way to late-convert a param f(y?) containing a VarRef into the alias you would have had if you had received the param as f(&y). Normally, I'd write a short utility function for it, but that doesn't seem possible here.

Maybe there are a couple of options for streamlined syntax that wouldn't involve the misleading fn(&var?) syntax discussed earlier:

1. Operator approach. Ideally such an operator would be parallel in syntax to the new ??= operator. As I understand it, y ??= x means "only evaluate the rhs x and assign to y if y is unset, otherwise short-circuit past the rhs". It seems like what we're doing here is the parallel but opposite case: "assign to this varref if and only if the var containing this varref is set and actually contains a varref, else short-circuit and do not evaluate the rhs". I hate to propose syntax, as others are better at that, but maybe I'll try: y %%= x could be equivalent to the rather messy IsSet(y) && y is VarRef && %y%:=x.

2. Function approach. Could there be a function that late-converts the y in f(y?) to an alias, as if f(&y) had been the receiver... such that, after that point, derefs aren't used? So maybe Ref(&y) could change the y containing a varref into the alias to the caller's var, but only if IsSet(y) && y is VarRef, returning true if the conversion succeeded else false. Then the usage syntax would then look something like: Ref(&y) && y:=x. I still think option 1 is better, and I'm not sure this option 2 is even feasible given the scope implications.

@lexikos, could either of these work, and would you be willing? I always hesitate to even propose anything like this, as you've already done sooo much! It's just such a common pattern to return optionally byref (esp important where calcs are expensive). All your great changes have made the language so beautiful and streamlined in so many other ways. Thanks for your time and consideration.

[lexikos]

Did you see the option I posted in the other topic?

I implemented short-circuiting %a?% ?? b and (%a?% := b) ?? c, but reverted it because it was inconsistent with the policy of prohibiting (a?.b := c) ?? d. I copied that restriction from JavaScript and deferred thiking about it further. I'm not sure why JavaScript doesn't allow a?.b = c; it's not as though it would prohibit a?.Set(b, c) or other method calls which mutate an object. I guess it might have to do with doubt about what exactly is short-circuited, which might be less of a concern for AutoHotkey because of the restrictions applied to unset expressions.

It was also a bit questionable because the short-circuit passes through the enclosing percent signs, whereas optional chaining normally only skips member access (including evaluation of parameters). It could be possible to short-circuit basically any expression (like ((pi * r?) ** 2) ?? a, but I'm not sure of the ramifications of allowing it. One might be tempted to write an expression like a[b?] ?? c, expecting it to short-circuit the array indexer when it actually invokes a[unset], or vice versa.

Don't use is VarRef. I don't know if you noticed, but v2.1-alpha.10 allows &x.y and other virtual references which are not a VarRef.

Side note: you can define VarRef.Prototype.Set and use a?.Set(b, c), but your code won't work with virtual references.

What do you do when the caller passes in a value which is not a VarRef (or virtual reference)? (IsSet(y) && y is VarRef && %y%:=x*3) ignores the (presumably invalid) value, whereas (IsSet(y) && %y%:=x*3) throws an error.

[Descolada]

I implemented short-circuiting %a?% ?? b and (%a?% := b) ?? c, but reverted it because it was inconsistent with the policy of prohibiting (a?.b := c) ?? d. I copied that restriction from JavaScript and deferred thiking about it further. I'm not sure why JavaScript doesn't allow a?.b = c; it's not as though it would prohibit a?.Set(b, c) or other method calls which mutate an object. I guess it might have to do with doubt about what exactly is short-circuited, which might be less of a concern for AutoHotkey because of the restrictions applied to unset expressions.

I found a proposal to implement such syntax in Javascript and don't see a good reason not to allow it. If a property can be get with such syntax, and as you said a?.Set(b) is allowed, why not also allow the proposed syntax. Thus I fully support allowing (a?.b := c) ?? d, additionally I wouldn't mind a[item] ??= c (Issue 6 of the same proposal) and a?.b++. ++a?.b on the other hand I wouldn't implement.

Don't use is VarRef. I don't know if you noticed, but v2.1-alpha.10 allows &x.y and other virtual references which are not a VarRef.

I hadn't noticed and don't yet actively use v2.1-alpha. Taking virtual references into account it does make sense to use (IsSet(y) && %y%:=x*3), though I don't particularly like it nor the option you posted in the other topic as both are cryptic and obfuscate the issue they are solving.

One possible solution: A_ThisFunc could instead of returning the function name return an object which would contain extra information about the called function. A_ThisFunc.Name would instead return the name, and then A_ThisFunc.IsDefault(ParamIndex) could return true/false whether a ByRef was provided or not (also it could return whether for f(x:=2) the user called f() or f(2)). Currently this would be a breaking change, so A_ThisFuncInfo could be an alternative name. Other properties/methods could be added in the future.

[lexikos]

a[item] ??= c is already implemented.

both are cryptic and obfuscate the issue they are solving.

In the other topic, I recommended avoiding ByRef when you want to know whether the parameter was omitted, due to the inherent ambiguity. When I posted the option in the other topic, I reiterated that I think it is better to avoid ByRef.

For (IsSet(y) && %y%:=x*3), I cannot see how it is anything but straight-forward and direct. The value of parameter y directly corresponds to whatever value the caller supplied. "If parameter y was supplied a value" is the condition under which you want to perform the assignment, and IsSet has been the most direct answer to this since it was added (one of the reasons it was added). %y%:= is the syntax provided for assigning to a reference. I fail to see how it is cryptic or obfuscating anything.

A_ThisFunc could instead of returning the function name return an object which would contain extra information about the called function.

In some cases, it would be useful to have a reference to the object which was actually called by the script - such as a BoundFunc - but that wouldn't be what you're asking for. What you want is information about the function call, not "this function" itself.

A_ThisFunc.IsDefault(ParamIndex) could return true/false whether a ByRef was provided or not

I am incredulous that you would suggest this after calling the solution above cryptic.

Keep in mind that you would be reading A_ThisFunc.IsDefault(3) or similar. You could at least have suggested to permit the parameter name.

Also, see reason #2 and reason #4.

Reason #2: You already have an elegant way to handle optional parameters of any type.

If you want to know whether a value was provided for a parameter, just declare it as p:=unset or p? and use IsSet(p).

Reason #4: Your suggestion requires that the program retains information about what parameter values were provided, when there is no guarantee that the implementation would need to retain that information for any other purpose. Hypothetically, by the time your checks execute, the parameter has been bound either to a reference provided by the caller or to a new local variable. At this point, it is just a name bound to a variable.

Could there be a function that late-converts the y in f(y?) to an alias

You could just use another user-defined function. I meant to mention this before, but forgot where I posted it:

Code: Select all

#Requires AutoHotkey v2.1-alpha.4
F(ref) {
    IsSet(ref) && ((&alias) {
        alias ??= 0
        alias++  ; Multiple references, no need for explicit %dereferencing%.
        alias++
        alias++
    })(ref)
}
F(&G)  ; G is unset
MsgBox G
F(&G)  ; G is not unset
MsgBox G

There can be other patterns, like making F itself an arrow function and making the main body an anonymous function, or encapsulating some of the logic in a separate function (which you can pass the ref and anonymous function) instead of putting (ref) at the end. F can have other parameters which don't need to be passed explicitly to the nested function.

AutoHotkey_H has long had a function that makes a variable an alias of another variable. I idly considered adding something, but didn't think of any good syntax. I think that allowing it might impose a constraint on future implementations. There are other ways to achieve the end result anyway.

[Descolada]

For (IsSet(y) && %y%:=x*3), I cannot see how it is anything but straight-forward and direct. The value of parameter y directly corresponds to whatever value the caller supplied. "If parameter y was supplied a value" is the condition under which you want to perform the assignment, and IsSet has been the most direct answer to this since it was added (one of the reasons it was added). %y%:= is the syntax provided for assigning to a reference. I fail to see how it is cryptic or obfuscating anything.

You proposed in an other thread the following option:

Code: Select all

a(&r := defaulted() => 0) {
	MsgBox "a: r was " ((r ?? 0) = defaulted ? "omitted" : "specified")
}

Lets say the function is part of a library and I'm trying to understand what the function does (perhaps from the Intellisense suggestion). Without reading any documentation it is only clear that the function can modify the input variable r, but it is not obvious what happens when a variable is not provided. Maybe a does something with the function object, or maybe it means the variable can also be a function object which will be called and the result used for something? The function definition is not self-explanatory, and in my opinion it doesn't look elegant.

OK, how about the other option:

Code: Select all

f(x, y?) => ((IsSet(y) && %y%:=x*3), x*2)

It is clear that y can be omitted, in which case the user could assume that some part of the code that would use y is skipped. Of course it also tells us that y won't be modified, because otherwise the function definition would be f(x, &y?), in line with built-in functions such as MouseGetPos: it is a simple rule that if there is a & then the variable will probably be modified, and if not then it shouldn't be modified. AFAIK such a rule isn't documented anywhere, but all built-in methods suggest that it exists. But instead we have two surprises here: first, the function actually accepts a VarRef or virtual reference and throws an error if a regular value is provided (hopefully the user deduces the cause of the "Variable not found" error!); second, it modifies the input variable y.

I might have been too harsh when calling this cryptic, because it could be written more clearly:

Code: Select all

f(x, y?) {
    if IsSet(y) { ; something was provided
        %y% := x*3 ; probably should be in a try..catch block to rethrow a clearer error message that a ByRef value was expected if a regular value was passed
    }
    return x*2
}

But what I would like is for both the function definition and the body of the function to be as self-explanatory as possible:

Code: Select all

f(x, &y?) {
    if !A_ThisFunc.IsDefault(2) { ; a VarRef/virtual reference was provided
        y := x*3
    }
    return x*2
}

In some cases, it would be useful to have a reference to the object which was actually called by the script - such as a BoundFunc - but that wouldn't be what you're asking for. What you want is information about the function call, not "this function" itself.

If the reference to the object would have a property or method to access the same information then I have no qualms about that, in fact I like it even more. Would be analogous to the this parameter in classes.

Keep in mind that you would be reading A_ThisFunc.IsDefault(3) or similar. You could at least have suggested to permit the parameter name.

I'd very much like to use the parameter name instead, but recommended the index number to be in line with the already existing FuncObj.IsByRef(ParamIndex?). If a name was used I would prefer A_ThisFunc.IsDefault(y) syntax to A_ThisFunc.IsDefault("y") so that if I decide to change the name of the parameter in the future then the interpreter warns me of an unset variable...

AutoHotkey_H has long had a function that makes a variable an alias of another variable. I idly considered adding something, but didn't think of any good syntax. I think that allowing it might impose a constraint on future implementations. There are other ways to achieve the end result anyway.

Doesn't seem to be too necessary (using objects largely gets the same result), but if it were to be implemented then perhaps x %%= &y? A combination of dereferencing and assignment syntax seems logical.

[lexikos]

You proposed in an other thread the following option:

Perhaps I was not clear: I do not recommend the option in that other thread. Given that I do not recommend it, I didn't care to say whether I agree or disagree with your assessment of that option. In the first place, I only posted it to show that it is possible to determine without ambiguity whether an optional ByRef parameter defaulted.

The paragraph you quoted does not relate to the option in the other thread at all. That is why it is a separate paragraph.

Of course it also tells us that y won't be modified

This is clearly a false assumption. Even putting aside VarRef, objects can be mutated without ByRef. Furthermore, the caller has to know what the parameters are for and what to pass, whether that may be a VarRef or a value. If the caller is passing a VarRef, the possibility that it could be assigned a value is obvious. If the caller is not passing a VarRef, they are using the function incorrectly.

My argument against the value of & as self-documenting is the same as the argument I gave in the other topic against the value of the automatic type-checking that it provides. Parameters which accept VarRef are only a fraction of all function parameters. Every other parameter type must be validated manually and documented separately. The self-documenting and self-validating properties of ByRef are just nice side-effects, not the purpose of the syntax.

Anyway, I didn't previously consider that you would consider the absence of & in the parameter definition as such a problem for clarity, but I still disagree. Aren't non-ByRef parameters cryptic and obfuscating, because they don't have any declarative syntax to show what the parameter accepts? It could be absolutely Anything.

In practical terms, I don't believe for a second that having or not having & visible in the parameter list makes a significant difference for the usability of the function.

For self-documenting and self-validating parameters, type annotations are a more generally applicable solution which I have considered. The name : type syntax which is used for typed properties could also be used for this.

... if not then it shouldn't be modified.

A variable can't be modified if you don't pass a reference. You have to pass a reference for the function to work.

Whether you declare the parameter ByRef or not, if you pass a VarRef, there is no guarantee that it will or will not be assigned a value. Clearly, user-defined functions aren't required to assign to a ByRef parameter and can use it for input/output, or just input (e.g. avoiding the copying of large strings). Using the & operator on a & parameter produces the original VarRef, so the parameter can even be used just for validation, with the VarRef itself never being dereferenced, only being stored for later use or returned to the caller (e.g. AssertIsRef(&r) => &r).

... but all built-in methods suggest that it exists.

I think you are conflating the recent use of & in the documentation with the age-old OutputVar convention, which is indicated by the parameter name and further documentation. There's also DllCall, which allows the use of VarRefs for input and output.

But what I would like is for both the function definition and the body of the function to be as self-explanatory as possible

What is x supposed to be? Use descriptive parameter and function names! And comments!

If the reference to the object would have a property or method to access the same information then I have no qualms about that

No, it would not. As I said, you want information about a function call, not about the function. There can be multiple calls to a function active at any given time. Although you would presumably want to know about the "top-most" one, attaching this information to the function itself is semantically unsound. On the more practical side, it assumes that multi-threading (or lesser substitutes like coroutines) will never be possible.

Doesn't seem to be too necessary (using objects largely gets the same result),

I was addressing the idea of "a function that late-converts the y in f(y?) to an alias", which is literally the same thing as a ByRef parameter, but with the VarRef being provided mid-function rather than as a parameter value. I don't disagree about this syntax sugar being unnecessary - so is ByRef - but that is beside the point.

Already in the current alpha, a ByRef parameter can be bound not only to a variable/property/array element, but to a pair of getter/setter functions (via a virtual reference). In the same way that built-in variables like A_TickCount are dynamic, a ByRef parameter can calculate its value on-demand or perform some action when assigned. That could be extended to any variable (although statically defining Name => Expression might be better for some use cases). Sure, you can just call functions directly, but would you want to replace all of the built-in variables with functions? Maybe you would.