[v2.0.14-v2.0.15] The p argument of property_get in the debugger does not work

[zero-plusplus]

As per the subject, the p argument is not working correctly in v2.0.14 and v2.0.15.

I set max_children to 100 and tried with array and map with 300 child elements.
If the p argument is 0, I get the child elements correctly, but if it is 1, it returns only one child element named [100], and if it is 2 or more, it returns no child elements.

PS
The p argument was not working correctly in v2.0.13 too.
Here too, the problem occurred when the p argument was greater than or equal to 1, and the elements [1]-[100] were retrieved regardless of the value of the p argument.
If the p argument was 0, the elements <base>-[99] were retrieved.

---

Also, on a related separate topic, has the specification for numchildren returned by property_get changed in v2?
It seems to return the number of elements, excluding item accesses such as arr[1].

This specification is very useful in vscode's paging process, where <base> and instance fields can be displayed as they are, and only array elements can be collapsed.

However, numchildren seems to be set to a number one greater than the actual number of elements.
Since the array has only <base> except for item access, I would guess that numchildren is 1, but it is actually set to 2.

[lexikos]

v2.0.16 fixes the paging.

Are you saying that it is useful for numchildren to count only the properties and not the enumerated items?

The specification for numchildren is "optional attribute with number of children for the property". The DBGp documentation is typically sparse on details, but it is safe to assume that this should be equivalent to the number of <property/> elements which would be the children of a property on page 0 if there was no upper limit (max_children = ∞).

numchildren is currently incorrect in v2.0 because returning the actual count would require enumerating all of the items. The numchildren attribute is written to the response buffer before any items are enumerated, because the raw XML is written (appended) as they are enumerated. I considered some ways to send a "more correct" count:

• Enumerate all items once to get the count, write the property opening tag including numchildren, then write the child properties, which means enumerating the items again (from 1 to the last item in the requested page).
• Enumerate items before writing anything to the response buffer. Instead, push the items of the current page into a temporary array. This would be complicated by the need to mix object-properties and enumerated items in one page.
• Partial solution: Cheat for Array, Map, Gui, etc. where the count is easily determined, and the enumerator is known to be the standard one.
• Build the response using the MSXML document object model, and serialize and write to the socket when complete. I might try this at some point to compare code size and performance, but the amount of work and risk of breaking something outweighs the apparent practical benefit of "fixing" numchildren. It would likely imply much greater use of dynamic memory; currently the debugger can reuse a single sufficiently-sized buffer for many XML packets.
• Build the child property XML separately, then insert into the response buffer. I only just thought of this. It should be simpler and lower risk because the existing code doesn't care which buffer it is writing into or whether a parent element was in there.

Problems with these solutions include:

• Even if there is only one __Enum call and one pass through the items, there would be more overhead just for the purpose of numchildren. Without numchildren, only the last page requires enumerating to the end, and pages which contain only properties would not even require the __Enum call.
• User-defined enumerators can enumerate infinitely, either due to a bug or by design. There would therefore still need to be an upper limit, which would likely be arbitrary. How many iterations should the debugger allow before deciding that it has done enough enumerating? Currently it is bounded by max_children.
• The debugger client still can't rely on numchildren being accurate with user-defined enumerators.
• While current versions are able to reuse the same response buffer for repeated XML responses, some of these solutions imply greater use of dynamic memory, which has a possible performance cost in addition to increasing heap fragmentation (in theory).

numchildren is currently calculated like ObjOwnPropCount(obj) + (ObjGetBase(obj) != "") + (HasMethod(obj, "__Enum") != 0).

The reason for the inclusion of __Enum is that if it is not a built-in function, an <enum> child property is written (and items are not automatically enumerated).

v2.1-alpha.10+ lists inherited properties as well, so the formula above is even more inaccurate. numchildren is removed entirely, which seems to be more correct than providing the wrong number. The DBGp documentation doesn't specify any case where the attribute is required.

This specification is very useful in vscode's paging process, where <base> and instance fields can be displayed as they are, and only array elements can be collapsed.

How is it used to achieve this? I guess that you might set max_children = numchildren and retrieve the properties with a single request. In practice, I don't think it would be much different to setting an arbitrary max_children, requesting more pages until all (or max) properties are retrieved, and discarding some items.

The DBGp specification doesn't appear to have any provision for custom attributes, but custom elements are explicitly permitted in errors and notifications, using XML namespaces to avoid conflicts. I think it would be safe to do the same for attributes, if any would be valuable enough to add.

[zero-plusplus]

Thanks for the fix, I have confirmed that the p argument works correctly.

---

Thanks for sharing your use about numchildren.
The reason for asking about numchildren is that to use paging with the DAP (Debug Adapter Protocol), the following values need to be set.

Code: Select all

// https://microsoft.github.io/debug-adapter-protocol/specification#Types_Variable

/**
 * The number of named child variables.
 * The client can use this information to present the children in a paged UI
 * and fetch them in chunks.
 */
namedVariables?: number;

/**
 * The number of indexed child variables.
 * The client can use this information to present the children in a paged UI
 * and fetch them in chunks.
 */
indexedVariables?: number;

When a named child variables is requested in the DAP's variablesRequest, I would hope that the following commands can be used to retrieve it.

Code: Select all

feature_set -n max_children -v numchildren
property_get -n variableName -p 0

As far as I have been able to ascertain, indexed child variables can be retrieved as follows.

Code: Select all

feature_set -n max_children -v 100
; When [0..99] is opened.
property_get -n variableName.<enum> -p 0
; When [100..199] is opened.
property_get -n variableName.<enum> -p 1

---

numchildren is currently calculated like ObjOwnPropCount(obj) + (ObjGetBase(obj) ! = “”) + (HasMethod(obj, “__Enum”) ! = 0).

The reason for the inclusion of __Enum is that if it is not a built-in function, an <enum> child property is written (and items are not automatically enumerated).

The <enum> is not actually displayed as a child element, so there was a problem with max_children = numchildren, which would get child elements that were not needed.

I can resolve this by using <enum> as numchildren - 1 if it is guaranteed that <enum> will not appear as a child element.

How is it used to achieve this? I guess that you might set max_children = numchildren and retrieve the properties with a single request. In practice, I don't think it would be much different to setting an arbitrary max_children, requesting more pages until all (or max) properties are retrieved, and discarding some items.

Named child variables like <base> and foo are enumerated before indexed child variables like `[1]`, right?

I know from experience that this is correct, but I don't know a definitive answer.

In vscode-autohotkey-debug v1.x.x I did not have to think about in which order the named variables are enumerated, as I get 10000 child variables by default, but when using paging it is necessary to get only the minimum number of child variables.

So I thought it would be useful if numchildren represented the number of named child variables and could be used to retrieve only named child variables.

[lexikos]

As far as I have been able to ascertain, indexed child variables can be retrieved as follows.

Named child variables like <base> and foo are enumerated before indexed child variables like `[1]`, right?

Yes, properties are enumerated before items.

The <enum> is not actually displayed as a child element

It depends on the type of __Enum, as I said.

I can resolve this by using <enum> as numchildren - 1 if it is guaranteed that <enum> will not appear as a child element.

There are several reasons not to do that:

• As I said, <enum> will appear as a child element (and the items will not) if __Enum is any object other than a built-in function.
• numchildren will not include <enum> at all if __Enum is not a method or a value property containing an object.
• numchildren may be changed to include <enum> only when it is going to be listed.
• numchildren has already been removed in v2.1-alpha.10, as I said.

when using paging it is necessary to get only the minimum number of child variables.

No, it might be more efficient to get only the DBGp properties that you require, but it is not necessary. You can always discard or ignore the DBGp properties you don't need, as with any other unnecessary information in the response.

Be aware that the current implementation does not cache any enumerated items or enumerators, so to retrieve a page it must enumerate all items from 1 to the end of the page. In other words, -p 0 will enumerate one page, -p 1 will enumerate two pages but only return one, etc. Retrieving larger pages means fewer repeat enumerations, and also less protocol overhead/latency.

The same applies to properties in v2.1-alpha.10+, which uses the Props enumerator to merge own and inherited properties. Determining the correct count without enumerating would be virtually impossible, as the enumeration process omits properties which are not inherited due to an overriding property.

If you want to be able to request only properties (in the AutoHotkey sense, not the DBGp sense), I could add a way to do that. Are there still cases where you want both at once (as they are returned now)? What do you do for objects with non-sequential items (e.g. Map, Gui) or custom enumerators?

[zero-plusplus]

numchildren has already been removed in v2.1-alpha.10, as I said.

I had missed it.
This is not a problem as the numchildren formula above is used.

---

Be aware that the current implementation does not cache any enumerated items or enumerators, so to retrieve a page it must enumerate all items from 1 to the end of the page. In other words, -p 0 will enumerate one page, -p 1 will enumerate two pages but only return one, etc. Retrieving larger pages means fewer repeat enumerations, and also less protocol overhead/latency.

The processing cost of JavaScript converting the retrieved elements into objects is probably greater than the time it takes to enumerate them, so it may be better to retrieve the minimum necessary.

Writing this reminded me that most of the slow processing of variable views in vscode-autohotkey-debug v1.x.x is caused by previewing objects like { key: 'value' }.

This was because the max_depth needed to be set to 2 in order to show the preview of the child elements; if the max_depth was 1, the preview of the child elements could not be shown as follows.

Code: Select all

; max_depth: 1
obj: { foo: Object }
  foo: {}
    bar: "baz"

Code: Select all

; max_depth: 2
obj: { foo: Object }
  foo: { bar: "baz" }
    bar: "baz"

Paging can lower the value of max_children, which indirectly solves this problem but is not perfect.

---

If you want to be able to request only properties (in the AutoHotkey sense, not the DBGp sense), I could add a way to do that. Are there still cases where you want both at once (as they are returned now)?

I forgot that function evaluation is implemented and can be solved on the AutoHotkey side.
However, I think the functionality to implement paging is all there for now.

---

What do you do for objects with non-sequential items (e.g. Map, Gui) or custom enumerators?

The number of elements in the Map is treated as indexedVariables.
This means that in the Variables view they are displayed as follows.

Code: Select all

myMap: Map(300) { ["key-001"] =>"value-001", ["key-002"] =>"value-002", ... }
  <base>: Prototype
  [0..99]
    ["key-001"]: "value-001"
    ["key-002"]: "value-002"
    ...
  [100..199]
  [200..299]

---

The extent to which paging is applied should also be considered.

For example, whether to cover user-defined objects that inherit from Array or Map, or Array-like objects with length or Map-like objects with count.

Advanced users may use collection libraries other than Array or Map, and it would be useful if they could support paging.

However, rules are required, such as the need to define properties with predefined names, such as length and count, to get the number of elements.

Also, applying paging to objects that are not taken into account might create side effects, such as some properties not being displayed correctly.