wow! what a beautiful trick; u managed to inline a loop cycle! and performance-wise its NOT getting worse. I learn alot from u =)oif2003 wrote: ↑12 Dec 2018, 18:37I knew there's a better way to write that indent function:Code: Select all
indent(n) => n ? indent(--n) " " : ""
AHK v2: converting/optimizing scripts Topic is solved
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: AHK v2 scripts optimization
Last edited by vvhitevvizard on 12 Dec 2018, 19:55, edited 3 times in total.
Re: Problems with objects and legacy syntax
Thanks! But interestingly, performance gets worse for me using recursion. The slow down over 20k runs ranges from almost none to up to 200 ms
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
U R RIGHT!
I re-tested it. Indeed, ~400 w/o it, 500 with it. Beautiful line of code, but terrible at run time ))
Last edited by vvhitevvizard on 12 Dec 2018, 19:08, edited 1 time in total.
Re: Problems with objects and legacy syntax
another static variable to keep track of last used this.Space and a quick check seems to fix it
Code: Select all
static q:= Chr(34), ind, space
(_d || space == this.Space) || (ind := indent(this.Space), space := this.Space)
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
It cannot but for json to/from conversions its ok. Ultimately It should be a user callback function. the same way replacer()/reviver() do it for its JS counterpart.
btw I realized I used different loop counts so I re-benched all the variants. Original is 500 ms. Mine with a closure gives +7%, ur 1st variant gives +15%, ur last variant (375 ms) gains +25%!
Last edited by vvhitevvizard on 12 Dec 2018, 19:28, edited 1 time in total.
Re: Problems with objects and legacy syntax
Cool! And you guessed it, my callback function is... format("{:g}", v) !vvhitevvizard wrote: ↑12 Dec 2018, 19:24It cannot but for json to/from conversions its ok. Ultimately It should be a user callback function. the same way replacer()/reviver() do it for its JS counterpart.
btw I re-benched all the variants. Mine with a closure gives +7%, ur 1st variant gives +15%, ur last variant gains +25%!
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
as an afterthought, Round() expects a pure float (tries to convert to float) and outputs a string. So its oke'ish, w/o preceding zeroes and numbers like 9..
All we need to do is make sure that (v is "Number") (is convertible to a number) before calling Round()
All we need to do is make sure that (v is "Number") (is convertible to a number) before calling Round()
Last edited by vvhitevvizard on 13 Dec 2018, 06:09, edited 3 times in total.
Re: Problems with objects and legacy syntax
When I tried that it gave me a lot of garbage after the last significant digit. like 102.000000097vvhitevvizard wrote: ↑12 Dec 2018, 19:37as an afterthought, Round() expects a pure float. So its output is oke'ish. w/o preceding zeroes and .. All we need to do is make sure that if (v is "Number") && Type(v)="String" -> v+=0 to force it pure number again
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
I edited my prev. post. We don't need to v+=0 for stringified output. We needed that conversion to a pure float when parsing stringified input.
and to the post before: so instead of having any callback for float parsing, trade-off would be Round(v, this.Round). where .Round is a public (not private) property for decimals N.
ofc its reasonable to process floats for stringified output and discard insignificant decimals (depends on the number's scale) to save space, like I do for 2 different cryptocurrencies when dumping price/volume numbers as .CSV text
1.
Code: Select all
0.0853;0.095;0.0853;0.00113
0.0853;0.0853;0.0853;0.00179
0.0853;0.0854;0.0853;0.00769
Code: Select all
4675;4700;4626;96610
4662;4695;4645;34720
4639;4689;4603;9713
Re: Problems with objects and legacy syntax
Sounds like a fine plan to me, but I'm sticking with my format(..., v)!vvhitevvizard wrote: ↑12 Dec 2018, 19:43We needed that conversion to a pure float when parsing stringified input.
and to the post before: so instead of having any callback for float parsing, trade-off would be Round(v, this.Round).
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
for json.get, ur version gains some performance, too. 1157 -> 1130 +2.4%
1.
Just 1 line that does the magic solely. Surprisingly, its inline "ifnot()" + ,
(!InStr(z, c)) && this.E(c, n)
a:=(b:=s.1).__Arr
->
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr
check it for urself!
this line executes for every token so inlining performance boost becomes tangible! Or we get different AHK pseudo-code as a result here.
It got me interested and I tested other 10+ changes. All the other json.get inlinings altogether could gain something but alas they end up with too small difference to be distinguishable. All the other changed stuff seems to be compiled to pseudo-code the same way. e.g. it doesn't matter whether expressions r in parentheses or not: z:=(kf:=(!a) && c==",") ? q:x vs z:=(kf:=!a && c==",") ? q : x is the same run-time speed -wise unless execution order changes. I honestly try to use parentheses just to comply with c-like syntax (AHK scripts to be a tad more easy to port in the future). Thats why Loop Parse drives me mad - I can write it neither Loop Parse(arguments*) nor Loop(Parse, arguments*)
2.
there is a glimpse of another possible tangible optimization, the line that executes for every token:
a:=(b:=s.1).__Arr
and it should be executed only for 2 conditions (InStr("}]", c)) and (InStr("{[", c)) cuz only these change stack s. But moving it away from the main cycle results in wrong data parsing. Ill try it later.
3.
Issue of moving throw(Exception()) to nested sub-functions is we have to change its second parameter (-2 now) to point at the first function outside json class.
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
1.
Just 1 line that does the magic solely. Surprisingly, its inline "ifnot()" + ,
(!InStr(z, c)) && this.E(c, n)
a:=(b:=s.1).__Arr
->
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr
check it for urself!
this line executes for every token so inlining performance boost becomes tangible! Or we get different AHK pseudo-code as a result here.
It got me interested and I tested other 10+ changes. All the other json.get inlinings altogether could gain something but alas they end up with too small difference to be distinguishable. All the other changed stuff seems to be compiled to pseudo-code the same way. e.g. it doesn't matter whether expressions r in parentheses or not: z:=(kf:=(!a) && c==",") ? q:x vs z:=(kf:=!a && c==",") ? q : x is the same run-time speed -wise unless execution order changes. I honestly try to use parentheses just to comply with c-like syntax (AHK scripts to be a tad more easy to port in the future). Thats why Loop Parse drives me mad - I can write it neither Loop Parse(arguments*) nor Loop(Parse, arguments*)
2.
there is a glimpse of another possible tangible optimization, the line that executes for every token:
a:=(b:=s.1).__Arr
and it should be executed only for 2 conditions (InStr("}]", c)) and (InStr("{[", c)) cuz only these change stack s. But moving it away from the main cycle results in wrong data parsing. Ill try it later.
3.
Issue of moving throw(Exception()) to nested sub-functions is we have to change its second parameter (-2 now) to point at the first function outside json class.
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
Last edited by vvhitevvizard on 12 Dec 2018, 23:19, edited 5 times in total.
Re: Problems with objects and legacy syntax
Interesting.... didn't realize you are still working on it. I got bored after dinner and tried this: about 10% faster for json.get
Two things I want to try: 1)look for replacement of regexmatch (~=), 2) try "loop parse _s" instead of using a while loop
Edit: item 2 won't work (or is too much work) since index n is changed multiple times inside the while loop
Code: Select all
#SingleInstance force
class json { ;json.set by vvhitevvizard. json.get by coco and modified by vvhitevvizard
get(ByRef _s) { ;obj:=json.get(str)
n:=0, k:="",kf:=0, s:=[root:={}], z:="{", static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
while c := SubStr(_s, ++n, 1)
InStr(" `t`n`r", c) || (
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}") : (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1), n += StrLen(v)+1, (kf) && (k:=v, z:=":", skip := true)) ;string literals
: (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1), n += StrLen(v)-1, (v is "Number") ? v += 0 : E(c, n)) ;number
,(skip) || (a ? k:=b.Push(v) : b[k]:=v, z := a ? ",]" : ",}"))
)
return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
}
static Compact:=0, Space:=4, ErrObjType:=0, ErrObj:=0
set(_o, _d:="") { ;_d:current indent
static q:= Chr(34), ind, space
if !IsObject(_o) ;"string" | number
return this.ErrObj && this.fthrow(Exception("Not an object.", -1))
(_d || space == this.Space) || ind := indent(space := this.Space), (this.Compact) ? n:=d:="" : n:="`n", d:=_d ind, VarSetCapacity(s,1024*8)
try a:=_o.__Arr ;trying to get a unique key
catch ;"Unknown property" exception means its a built-in inenumerable object
return (t:=Type(_o) " Object", (this.ErrObjType) && this.fthrow(Exception(t " type is not supported.", -1)), "{" q t q "}")
for i in _o ;due to the use of a for-loop, arrays such as '[1,,3]' are detected as objects({})
if !(a:=i=A_Index) ;a=0:associative or sparse array ([1,,3])
break
for k, v in _o ;recursive ;JSON standard: keys are always "strings", values r: number|"string"|{object}|[array]
s .= d (a ? "" : q k q ":") (isObject(v) ? this.set(v, d) : (Type(v)=="String") ? q v q : (Type(v)=="Float") ? format("{:g}", v) : v) "," n
return (a ? "[" : "{") (s ? (n RTrim(s, "," n) n _d) : "") (a ? "]" : "}") ;wrap to brackets
indent(n) => n ? indent(--n) " " : ""
}
fthrow(e) {
throw e
}
}
; Test runs
j:='{"r1":{"r2":{"type1":11,"type2":11},"misc":12}, "type":"ask","price":104.2,"amount":101, "a":[111,"aa"]}'
t:=A_TickCount
loop(20000)
b:=json.get(j)
a1:=A_TickCount-t
t:=A_TickCount
loop(20000)
s:=json.set(b)
a2:=A_TickCount-t
msgbox(clipboard:=a1 "|" a2 "`n`n" s)
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
I have a bad habit of switching between tasks... thats why I counterproductive haha
ur new version scores 940 vs 1160 +18.5% compared to the 1st version! Very impressive! U keep on amazing me. So inlining for routines' main cycle makes sense. Thou it becomes somewhat less editable.
but it gives an error on real data. 1MB file from the server. Ill try to pinpoint the issue
and ur lines become way too long to my taste, I changed it a bit. same perf:
Code: Select all
get(ByRef _s) { ;obj:=json.get(str)
n:=0, k:="",kf:=0, s:=[root:={}], z:="{"
static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
while c := SubStr(_s, ++n, 1)
InStr(" `t`n`r", c) || (
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}")
: (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1)
, n += StrLen(v)+1, (kf) && (k:=v, z:=":", skip := true)) ;string literals
: (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1)
, n += StrLen(v)-1, (v is "Number") ? v += 0 : E(c, n)) ;number
,(skip) || (a ? k:=b.Push(v) : b[k]:=v, z := a ? ",]" : ",}"))
)
return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
}
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
a:=(b:=s.1).__Arr shouldnt be in the main cycle
Last edited by vvhitevvizard on 12 Dec 2018, 22:42, edited 1 time in total.
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
a fragment of real world data that triggers an error:
j:='{"btc_usd":[{"type":"bid","price":3665,"amount":0.00014128,"tid":202998772,"timestamp":1544493122}]}'
->
Unexpected char <0> at pos 49
(thats within "amount":0.00014128 token)
ur previous variant (I mean ur changes I kept for myself just to gain the perf boost) works ok with the json data above
here it goes:
btw I see uve changed ur mind and returned indent(n) => n ? indent(--n) " " : ""
instead of nasty-looking
Performance of json.set reacts upon this negatively. =)
j:='{"btc_usd":[{"type":"bid","price":3665,"amount":0.00014128,"tid":202998772,"timestamp":1544493122}]}'
->
Unexpected char <0> at pos 49
(thats within "amount":0.00014128 token)
ur previous variant (I mean ur changes I kept for myself just to gain the perf boost) works ok with the json data above
here it goes:
Code: Select all
get(ByRef _s){ ;obj:=json.get(str)
static q:=Chr(34)
static x:=q "{[0123456789-" ;json allowed chars outside strings
static p:={"__Arr":1}
n:=0, k:="",kf:=0, s:=[root:={}], z:="{"
while((c:=SubStr(_s, ++n, 1))!==""){
if(InStr(" `t`n`r", c))
continue
(!InStr(z, c)) && this.E(c, n)
a:=(b:=s.1).__Arr
if(InStr("}]", c)){
(s.1=root) && this.E(c, n)
s.RemoveAt(1), z:=s.1.__Arr ? ",]":",}"
}
else if(InStr(",:", c))
z:=(kf:=(!a) && c==",") ? q:x
else if(InStr("{[", c)){
(c=="{") ? (kf:=1, v:={}, z:=q "}"):(v:=new p, z:=x "]"), s.InsertAt(1,v)
a ? k:=b.Push(v):b[k]:=v
}
else{
if(c==q){ ;string literals
v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1), n+=StrLen(v)+1
if(kf){
k:=v, z:=":"
continue
}
}
else{ ;number
v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1), n+=StrLen(v)-1
(v is "Number") ? v+=0:this.E(c, n)
}
a ? k:=b.Push(v):b[k]:=v, z:=a ? ",]":",}"
}
}
(s.1!=root) && this.E(c, n) ;s.Count()!=1 ;unpaired {}, []
return b
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
}
instead of nasty-looking
Code: Select all
indent(n){
loop(n)
_.=" "
return _
}
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
I found a culprit: while c := SubStr(_s, ++n, 1)
we compare the current stream char with "" aka EOF. Proper previous code should be kept here: while((c:=SubStr(_s, ++n, 1))!=="")
So the latest functioning json.get method is:
oh and I changed skip to u and new booleans u added to 0/1. just for style )
Overall its a state of programming art. But I feel there is a room for performance improvement. btw unlike the original coco version, I made sure it catches all the errors but simplified it to just pinpoint error char and position.
we compare the current stream char with "" aka EOF. Proper previous code should be kept here: while((c:=SubStr(_s, ++n, 1))!=="")
So the latest functioning json.get method is:
Code: Select all
;json.get by coco and heavily optimized by vvhitevvizard and oif2003
get(ByRef _s) { ;obj:=json.get(str)
n:=0, k:="",kf:=0, s:=[root:={}], z:="{"
static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
while((c:=SubStr(_s, ++n, 1))!=="")
;while c := SubStr(_s, ++n, 1)
InStr(" `t`n`r", c) || (
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, u:=0
, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}")
: (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1)
, n+=StrLen(v)+1, (kf) && (k:=v, z:=":", u:=1)) ;string literals
: (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1)
, n+=StrLen(v)-1, (v is "Number") ? v+=0:E(c, n)) ;number
,(u) || (a ? k:=b.Push(v) : b[k]:=v, z:=a ? ",]" : ",}"))
)
return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
E(_c,_n)=>this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -2))
}
Overall its a state of programming art. But I feel there is a room for performance improvement. btw unlike the original coco version, I made sure it catches all the errors but simplified it to just pinpoint error char and position.
Last edited by vvhitevvizard on 12 Dec 2018, 23:45, edited 1 time in total.
Re: Problems with objects and legacy syntax
That was my fault ... I removed that !== ""
I think we might be able to replace
Code: Select all
(SubStr(_s, n) ~= "[\]\},\s]|$")-1
Code: Select all
min(InStr(_s,"]",,n)||M,InStr(_s,"}",,n)||M,InStr(_s,",",,n)||M,InStr(_s," ",,n)||M)-n
Took me a while, but that's the best I can come up with, it is marginally faster on my tests. Almost not worth the extra characters... but at least I can declare victory over RegExMatch for this function.
Re: Problems with objects and legacy syntax
I changed it back because I figure after the static checks the overhead should be constant unless you change this.Space every other callvvhitevvizard wrote: ↑12 Dec 2018, 22:40btw I see uve changed ur mind and returned indent(n) => n ? indent(--n) " " : ""
instead of nasty-lookingPerformance of json.set reacts upon this negatively. =)Code: Select all
indent(n){ loop(n) _.=" " return _ }
So right now I'm at:
Spoiler
- vvhitevvizard
- Posts: 454
- Joined: 25 Nov 2018, 10:15
- Location: Russia
Re: Problems with objects and legacy syntax
u can replace it with m:=2**31-1
but why 32 bits? x64 systems use 64 bit integer. 2**63-1. nvm 32 bits is enuf for ur task - it works )
With ur talent of replacing conditional expressions to unconditional ones u could be a great Streaming SIMD Extensions coder.
Ive tested it. 930 vs 950. +2.1%. +19.8% overall. and greatly reduced code that fits on 1 screen
but ur code would fail on \t or EOL symbols. any "beautified" json input would crash. and if u add more comparisons it would become slower than regex
actually our own beautified output would crash json.get. Try that sample as input data (AHK v2 only syntax):
Code: Select all
j:="
(
{
"a":[
111,
"aa"
],
"amount":101,
"price":104.2,
"r1":{
"misc":12,
"r2":{
"type1":11,
"type2":11
}
},
"type":"ask"
}
)"
Last edited by vvhitevvizard on 13 Dec 2018, 00:25, edited 15 times in total.
Who is online
Users browsing this forum: Google [Bot], mikeyww and 36 guests