People maintaining AutoHotkey support in editors would benefit from a standard source of syntax data.
Currently the most complete source is the documentation, as it must be kept up to date, and it includes its own syntax highlighter. The documentation's syntax definitions include parameter names, whereas AutoHotkey v1 itself doesn't include them in the program or source code, and v2 includes them for many functions, but only in the source code, and not yet all functions.
@fincs wrote some scripts that get basic keyword lists; I'm not sure whether he's made any further progress.
I started to write a script for scraping function definitions from the documentation, but it's really not something I want to spend much time. There are so many more interesting things I could be working on, like AutoHotkey v2.1.
So I'm starting this topic in the hope that there will be volunteers to take up this project.
Goals:
Produce a script (or scripts) for v2 that scrapes both v1 and v2 documentation.
Scrape keyword lists (easy).
Scrape parameter definitions for functions, directives and v1 commands.
Exclude control flow statements, since they can be unique.
Produce maps or arrays of objects, which tool authors can iterate over to dump data in an appropriate format.
Optional: include a function for dumping the data in standard formats such as JSON and XML.
Once it is in a working state, it should be submitted to the documentation repository and kept in sync with the documentation.
My script is in a very rough state. I would not suggest using it as a base directly, but perhaps it will be of some use for reference.
#include <ActiveScript>
js := ActiveScript('JScript')
isV1 := false
subcmdNumber := isV1 ? 7 : -1
propertyNumber := isV1 ? 8 : 7
if isV1
SetWorkingDir '..\v1'
for r in [&directives, &variables, &functions, &statements, &commands, &keywords, &properties]
%r% := Mapi()
index := js.Eval(FileRead('docs\static\source\data_index.js'))
loop index.length {
item := index.%A_Index-1%
if item.length < 3
continue
switch item.2 {
case 0: ; directive
directives[item.0] := item.1
case 1: ; variable
variables[item.0] := item.1
case 2: ; function
functions[StrReplace(item.0, '()')] := item.1
case 3: ; control flow
if !IsAlpha(item.0) ; exclude "Catch ... as"
continue
statements[item.0] := item.1
case 4, 5: ; operator, declaration keyword
if !IsAlpha(item.0) ; exclude symbols and "class ... extends"
|| item.0 = "ByRef" && !isV1
continue
keywords[item.0] := item.1
case 6:
if isV1 ; command
commands[item.0] := item.1
else ; class
variables[item.0] := item.1
case subcmdNumber: ; sub-command
keywords[item.0] := item.1
case propertyNumber: ; method/property
if !RegExMatch(item.0, '^(\w+) \S+ \((\w+)\)$', &m) {
D "unrecognized type 7 index entry: " item.0
continue
}
(c := properties.Get(m.1, 0)) || properties[m.1] := c := Mapi()
c[m.2] := item.1
}
}
; lacking Syntax: ObjFromPtr, ObjFromPtrAddRef, ObjHasOWnProp, ObjOwnProps
; , ObjPtr, ObjPtrAddRef
; special cases:
; StrPut, RegWrite - overloads
; IsXxx
for item, uri in functions {
html := FileRead('docs\' RegExReplace(uri, '#.*'))
if RegExMatch(html, 'im)(?:<pre class="Syntax">\s*|^)(?:(\w+) := +)?<span class="func">' item '</span>\K.*?(?=</pre>|(?<!</pre>)$)', &m) {
returns := m.1
syntax := m.0
syntax := RegExReplace(syntax, '</?[ai]\b.*?>|<em>;.*?</em>')
syntax := StrReplace(syntax, '&', '&')
params := [], is_opt := false
i := 1 + (SubStr(syntax,1,1) == '(')
while i := RegExMatch(syntax, '<span class="optional">(*:opt)|</span>(*:req)'
. '|,? *(?<ref>&)?(?<id>[\w-\\#"\.\?]+)(?> *:= *+(?<def>[^,)<\]]*))?|[^, ()](*:err)', &m, i) {
; FIXME: allowing ? for some parameter names in v1
i += m.Len
switch m.Mark {
case "opt": is_opt := true
case "req": is_opt := false
case "err":
D "parse error: " m.0 "`n" item syntax
continue 2
default:
pair := StrSplit(m.0, ":=", " ", 2)
params.Push {
Name: m.id,
IsByRef: m.ref != "",
IsOptional: is_opt || m.def != "",
Default: m.def
}
}
}
; syntax := RegExReplace(syntax, '<span class="optional">(.*?)</span>', '[$1]')
; syntax := item (SubStr(syntax,1,1) = ' ' ? '(' SubStr(syntax,2) ')' : syntax)
; syntax := item syntax
; (returns) && syntax .= ' => ' returns
; D syntax
syntax2 := ''
for param in params
syntax2 .= (
(param.IsByRef ? '&' : '')
(param.Name)
(param.Default != '' ? ' := ' param.Default : param.IsOptional && !InStr(param.Name, '...') ? '?' : '')
(A_Index < params.Length ? ', ' : '')
)
if isV1 && !returns ; FIXME: differentiate commands and functions better
; syntax2 := item ' ' syntax2
syntax2 := item '(' syntax2 ')'
else
syntax2 := item ((params.Length || returns) ? '(' syntax2 ')' : '')
(returns) && syntax2 .= ' => ' returns
D syntax2
if !isV1 && (f := %item%?, IsSet(f)) {
if po := f is Class ? 1 : 0
f := f.Call
if f.MaxParams - po > params.Length
D '** PARAM COUNT MISMATCH: ' params.Length ' should be ' (f.MaxParams - po)
for param in params {
if param.IsByRef != f.IsByRef(A_Index + po)
D '** BYREF MISMATCH: ' A_Index ' ' param.Name
if param.IsOptional != f.IsOptional(A_Index + po)
D '** OPTIONAL MISMATCH: ' param.Name (param.IsOptional ? '?' : '')
}
}
; else D '** UNDEFINED: ' item
}
else if item ~= 'i)^Is(Alpha|Alnum|Upper|Lower)$'
D item '(String, Mode?) => Boolean'
else if uri ~= '/Is.htm#'
D item '(Value) => Boolean'
else
D '** ' item ' ' uri
; break
}
; fixme: ControlSendText: Same parameters as above
; fixme: DllCall, ComObjType
Exit
D(s) {
try
FileAppend s "`n", "*"
catch
OutputDebug s "`n"
}
class Mapi extends Map {
CaseSense := 'off'
}
Some notes:
The script is expected to be in the root directory of a clone of the v2 documentation repository, with a v1 directory in the parent directory if isV1 is set to true.
I started out using eval for the index data, but later realized that it would be so trivial to parse with RegExMatch that it is probably better to do so, removing the dependency on ActiveScript/ScriptControl/MSHTML.
I started out trying to use MSHTML to parse the HTML (ComObjGet or WebBrowser.Navigate), but something in the documentation sidebar scripts caused it to "html-encode" all of the HTML, utterly breaking it. Parsing it with RegEx instead wasn't difficult. An alternative would be to strip out the <script> elements and use HTMLDocument.write().
My script uses the optional? convention new to v2, just for debug output. There are some parameter names in v1 that have a "?" suffix with no particular meaning.
functions in for item, uri in functions can be changed to directives or commands to test those.
When !isV1, the script tries to verify the results by checking the properties of Func. There are several cases where I think it is not worthwhile to correct inconsistencies (e.g. Control is actually mandatory in most cases even when it is preceded by an optional parameter). It also picks up some cases where the scraping code is insufficient, because the function has multiple usages.
There are some cases where functions lack a <pre class="Syntax"> block in the documentation, or have multiple definitions or "overloads". It would be acceptable to special-case these within the script, or adjust the documentation.
I think I have done a some part of this job. For the convenience of my own extension of vs code (focus on v1 for now, and plan to shift to v2), I collected most of v1 functions and command from the forum builtin.ts. It is basically a big json in a typescript file.
And another file may be helpful. LangSpec A LangSpec covers part of v1 syntax.
There are other files defining v1 and v2 syntax for other projects. That is not the point. As per the first line of my post, it is about having a standard source of syntax metadata. If a new function is added, for instance, it will be added to the documentation and can then be scraped by a script to produce up-to-date syntax definitions. As an extension author, you can hypothetically use this to keep your definitions up to date.
This snippet scrapes the member of class. Instead of using IE which will execute javascript, htmlfile merely load html and everthing will remain the same. I think it is helpful.
