AHK v2: converting/optimizing scripts Topic is solved

[vvhitevvizard]

I knew there's a better way to write that indent function:

Code: Select all

indent(n) => n ? indent(--n) " " : ""

wow! what a beautiful trick; u managed to inline a loop cycle! and performance-wise its NOT getting worse. I learn alot from u =)

[oif2003]

Thanks! But interestingly, performance gets worse for me using recursion. The slow down over 20k runs ranges from almost none to up to 200 ms

[vvhitevvizard]

Thanks! But interestingly, performance gets worse for me using recursion. The slow down over 20k runs ranges from almost none to up to 200 ms

U R RIGHT!
I re-tested it. Indeed, ~400 w/o it, 500 with it. Beautiful line of code, but terrible at run time ))

[oif2003]

another static variable to keep track of last used this.Space and a quick check seems to fix it

Code: Select all

		static q:= Chr(34), ind, space
		(_d || space == this.Space) || (ind := indent(this.Space), space := this.Space)

[vvhitevvizard]

another static variable to keep track of last used this.Space and a quick check seems to fix it

400 -> 375

[vvhitevvizard]

But I don't think round(v, 8) handles trailing zeros?

It cannot but for json to/from conversions its ok. Ultimately It should be a user callback function. the same way replacer()/reviver() do it for its JS counterpart.
btw I realized I used different loop counts so I re-benched all the variants. Original is 500 ms. Mine with a closure gives +7%, ur 1st variant gives +15%, ur last variant (375 ms) gains +25%!

[oif2003]

But I don't think round(v, 8) handles trailing zeros?

It cannot but for json to/from conversions its ok. Ultimately It should be a user callback function. the same way replacer()/reviver() do it for its JS counterpart.
btw I re-benched all the variants. Mine with a closure gives +7%, ur 1st variant gives +15%, ur last variant gains +25%!

Cool! And you guessed it, my callback function is... format("{:g}", v) !

[vvhitevvizard]

my callback function is... format("{:g}", v) !

haha nice joke.
method should look like set(_o, _f:=""){ ;object, callback but calling out an outer function for any value would drastically reduce performance. I tested it

[vvhitevvizard]

as an afterthought, Round() expects a pure float (tries to convert to float) and outputs a string. So its oke'ish, w/o preceding zeroes and numbers like 9..
All we need to do is make sure that (v is "Number") (is convertible to a number) before calling Round()

[oif2003]

as an afterthought, Round() expects a pure float. So its output is oke'ish. w/o preceding zeroes and .. All we need to do is make sure that if (v is "Number") && Type(v)="String" -> v+=0 to force it pure number again

When I tried that it gave me a lot of garbage after the last significant digit. like 102.000000097

[vvhitevvizard]

When I tried that it gave me a lot of garbage after the last significant digit. like 102.000000097

I edited my prev. post. We don't need to v+=0 for stringified output. We needed that conversion to a pure float when parsing stringified input.

and to the post before: so instead of having any callback for float parsing, trade-off would be Round(v, this.Round). where .Round is a public (not private) property for decimals N.

ofc its reasonable to process floats for stringified output and discard insignificant decimals (depends on the number's scale) to save space, like I do for 2 different cryptocurrencies when dumping price/volume numbers as .CSV text
1.

Code: Select all

0.0853;0.095;0.0853;0.00113
0.0853;0.0853;0.0853;0.00179
0.0853;0.0854;0.0853;0.00769

2.

Code: Select all

4675;4700;4626;96610
4662;4695;4645;34720
4639;4689;4603;9713

PS: the topic should be renamed to "AHK v2 scripts optimization" =)

[oif2003]

We needed that conversion to a pure float when parsing stringified input.

and to the post before: so instead of having any callback for float parsing, trade-off would be Round(v, this.Round).

Sounds like a fine plan to me, but I'm sticking with my format(..., v)!

[vvhitevvizard]

for json.get, ur version gains some performance, too. 1157 -> 1130 +2.4%
1.
Just 1 line that does the magic solely. Surprisingly, its inline "ifnot()" + ,
(!InStr(z, c)) && this.E(c, n)
a:=(b:=s.1).__Arr

->
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr
check it for urself!
this line executes for every token so inlining performance boost becomes tangible! Or we get different AHK pseudo-code as a result here.

It got me interested and I tested other 10+ changes. All the other json.get inlinings altogether could gain something but alas they end up with too small difference to be distinguishable. All the other changed stuff seems to be compiled to pseudo-code the same way. e.g. it doesn't matter whether expressions r in parentheses or not: z:=(kf:=(!a) && c==",") ? q:x vs z:=(kf:=!a && c==",") ? q : x is the same run-time speed -wise unless execution order changes. I honestly try to use parentheses just to comply with c-like syntax (AHK scripts to be a tad more easy to port in the future). Thats why Loop Parse drives me mad - I can write it neither Loop Parse(arguments*) nor Loop(Parse, arguments*)

2.
there is a glimpse of another possible tangible optimization, the line that executes for every token:
a:=(b:=s.1).__Arr
and it should be executed only for 2 conditions (InStr("}]", c)) and (InStr("{[", c)) cuz only these change stack s. But moving it away from the main cycle results in wrong data parsing. Ill try it later.

3.
Issue of moving throw(Exception()) to nested sub-functions is we have to change its second parameter (-2 now) to point at the first function outside json class.
E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))

[oif2003]

Interesting.... didn't realize you are still working on it. I got bored after dinner and tried this: about 10% faster for json.get
Two things I want to try: 1)look for replacement of regexmatch (~=), 2) try "loop parse _s" instead of using a while loop
Edit: item 2 won't work (or is too much work) since index n is changed multiple times inside the while loop

Code: Select all

#SingleInstance force

class json { ;json.set by vvhitevvizard. json.get by coco and modified by vvhitevvizard
	get(ByRef _s) { ;obj:=json.get(str)
		n:=0, k:="",kf:=0, s:=[root:={}], z:="{", static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
		while c := SubStr(_s, ++n, 1)
			InStr(" `t`n`r", c) || ( 
				InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
				, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
				: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
				: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}") : (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
				: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1), n += StrLen(v)+1, (kf) && (k:=v, z:=":", skip := true)) ;string literals
						  : (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1), n += StrLen(v)-1, (v is "Number") ? v += 0 : E(c, n)) ;number
				 ,(skip) || (a ? k:=b.Push(v) : b[k]:=v, z := a ? ",]" : ",}"))
			)
		return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
		E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
	}
	static Compact:=0, Space:=4, ErrObjType:=0, ErrObj:=0 		
	set(_o, _d:="") { ;_d:current indent
		static q:= Chr(34), ind, space
		if !IsObject(_o) ;"string" | number
			return this.ErrObj && this.fthrow(Exception("Not an object.", -1))
		(_d || space == this.Space) || ind := indent(space := this.Space), (this.Compact) ? n:=d:="" : n:="`n", d:=_d ind, VarSetCapacity(s,1024*8)
		try a:=_o.__Arr ;trying to get a unique key
		catch ;"Unknown property" exception means its a built-in inenumerable object
			return (t:=Type(_o) " Object", (this.ErrObjType) && this.fthrow(Exception(t " type is not supported.", -1)), "{" q t q "}")
		for i in _o	;due to the use of a for-loop, arrays such as '[1,,3]' are detected as objects({})
			if !(a:=i=A_Index) ;a=0:associative or sparse array ([1,,3])
				break
		for k, v in _o ;recursive ;JSON standard: keys are always "strings", values r: number|"string"|{object}|[array]
			s .=  d (a ? "" : q k q ":") (isObject(v) ? this.set(v, d) : (Type(v)=="String") ? q v q : (Type(v)=="Float") ? format("{:g}", v) : v) "," n
		return (a ? "[" : "{") (s ? (n RTrim(s, "," n) n _d) : "") (a ? "]" : "}") ;wrap to brackets
		indent(n) => n ? indent(--n) " " : ""
	}
	fthrow(e) {
		throw e
	}
}

; Test runs
j:='{"r1":{"r2":{"type1":11,"type2":11},"misc":12}, "type":"ask","price":104.2,"amount":101, "a":[111,"aa"]}'

t:=A_TickCount
loop(20000)
	b:=json.get(j)
a1:=A_TickCount-t

t:=A_TickCount
loop(20000)
	s:=json.set(b)
a2:=A_TickCount-t
	
msgbox(clipboard:=a1 "|" a2 "`n`n" s)

[vvhitevvizard]

Interesting.... didn't realize you are still working on it. I got bored after dinner and tried this: about 10% faster for json.get

I have a bad habit of switching between tasks... thats why I counterproductive haha
ur new version scores 940 vs 1160 +18.5% compared to the 1st version! Very impressive! U keep on amazing me. So inlining for routines' main cycle makes sense. Thou it becomes somewhat less editable.

but it gives an error on real data. 1MB file from the server. Ill try to pinpoint the issue

and ur lines become way too long to my taste, I changed it a bit. same perf:

Code: Select all

	get(ByRef _s) { ;obj:=json.get(str)
		n:=0, k:="",kf:=0, s:=[root:={}], z:="{"
		static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
		while c := SubStr(_s, ++n, 1)
			InStr(" `t`n`r", c) || ( 
				InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
				, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
				: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
				: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}") 
					: (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
				: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1)
						, n += StrLen(v)+1, (kf) && (k:=v, z:=":", skip := true)) ;string literals
					: (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1)
						, n += StrLen(v)-1, (v is "Number") ? v += 0 : E(c, n)) ;number
				 ,(skip) || (a ? k:=b.Push(v) : b[k]:=v, z := a ? ",]" : ",}"))
			)
		return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
		E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
	}

pls pay attention to this line:
InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, skip := false
a:=(b:=s.1).__Arr shouldnt be in the main cycle

[vvhitevvizard]

a fragment of real world data that triggers an error:
j:='{"btc_usd":[{"type":"bid","price":3665,"amount":0.00014128,"tid":202998772,"timestamp":1544493122}]}'
->
Unexpected char <0> at pos 49
(thats within "amount":0.00014128 token)
ur previous variant (I mean ur changes I kept for myself just to gain the perf boost) works ok with the json data above
here it goes:

Code: Select all

	get(ByRef _s){ ;obj:=json.get(str)
		static q:=Chr(34)
		static x:=q "{[0123456789-" ;json allowed chars outside strings
		static p:={"__Arr":1}
		n:=0, k:="",kf:=0, s:=[root:={}], z:="{"

		while((c:=SubStr(_s, ++n, 1))!==""){
			if(InStr(" `t`n`r", c))
				continue
			(!InStr(z, c)) && this.E(c, n)
			a:=(b:=s.1).__Arr

			if(InStr("}]", c)){
				(s.1=root) && this.E(c, n)
				s.RemoveAt(1), z:=s.1.__Arr ? ",]":",}"
			}
			else if(InStr(",:", c))
				z:=(kf:=(!a) && c==",") ? q:x
			else if(InStr("{[", c)){
				(c=="{") ? (kf:=1, v:={}, z:=q "}"):(v:=new p, z:=x "]"), s.InsertAt(1,v)
				a ? k:=b.Push(v):b[k]:=v
			}
			else{
				if(c==q){ ;string literals
					v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1), n+=StrLen(v)+1
					if(kf){
						k:=v, z:=":"
						continue
					}
				}
				else{ ;number
					v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1), n+=StrLen(v)-1
					(v is "Number") ? v+=0:this.E(c, n)
				}
				a ? k:=b.Push(v):b[k]:=v, z:=a ? ",]":",}"
			}
		}
		(s.1!=root) && this.E(c, n) ;s.Count()!=1 ;unpaired {}, []
		return b

		E(_c,_n) => this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -1))
	}

btw I see uve changed ur mind and returned indent(n) => n ? indent(--n) " " : ""
instead of nasty-looking

Code: Select all

	indent(n){
		loop(n)
			_.=" "
		return _
	}

Performance of json.set reacts upon this negatively. =)

[vvhitevvizard]

I found a culprit: while c := SubStr(_s, ++n, 1)
we compare the current stream char with "" aka EOF. Proper previous code should be kept here: while((c:=SubStr(_s, ++n, 1))!=="")

So the latest functioning json.get method is:

Code: Select all

;json.get by coco and heavily optimized by vvhitevvizard and oif2003
	get(ByRef _s) { ;obj:=json.get(str)
		n:=0, k:="",kf:=0, s:=[root:={}], z:="{"
		static q:=Chr(34), p:={"__Arr":1} ,x:=q "{[0123456789-" ;json allowed chars outside strings
		while((c:=SubStr(_s, ++n, 1))!=="")
		;while c := SubStr(_s, ++n, 1)
			InStr(" `t`n`r", c) || (
				InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, u:=0
				, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
				: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
				: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}") 
					: (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
				: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1)
						, n+=StrLen(v)+1, (kf) && (k:=v, z:=":", u:=1)) ;string literals
					: (v:=SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1)
						, n+=StrLen(v)-1, (v is "Number") ? v+=0:E(c, n)) ;number
				 ,(u) || (a ? k:=b.Push(v) : b[k]:=v, z:=a ? ",]" : ",}"))
			)
		return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
		E(_c,_n)=>this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -2))
	}

oh and I changed skip to u and new booleans u added to 0/1. just for style )
Overall its a state of programming art. But I feel there is a room for performance improvement. btw unlike the original coco version, I made sure it catches all the errors but simplified it to just pinpoint error char and position.

[oif2003]

That was my fault ... I removed that !== ""

I think we might be able to replace

Code: Select all

(SubStr(_s, n) ~= "[\]\},\s]|$")-1

with

Code: Select all

min(InStr(_s,"]",,n)||M,InStr(_s,"}",,n)||M,InStr(_s,",",,n)||M,InStr(_s," ",,n)||M)-n

where M:=0x7FFFFFFF is declared as static inside get
Took me a while, but that's the best I can come up with, it is marginally faster on my tests. Almost not worth the extra characters... but at least I can declare victory over RegExMatch for this function.

[oif2003]

btw I see uve changed ur mind and returned indent(n) => n ? indent(--n) " " : ""
instead of nasty-looking

Code: Select all

	indent(n){
		loop(n)
			_.=" "
		return _
	}

Performance of json.set reacts upon this negatively. =)

I changed it back because I figure after the static checks the overhead should be constant unless you change this.Space every other call

So right now I'm at:

Spoiler

Code: Select all

#SingleInstance force

class json { ;json.set by vvhitevvizard. json.get by coco and modified by vvhitevvizard
	get(ByRef _s) { ;obj:=json.get(str)
		static M:=0x7FFFFFFF, q:=Chr(34), p:={"__Arr":1}, x:=q "{[0123456789-" ;json allowed chars outside strings
		n:=0, k:="", kf:=0, s:=[root:={}], z:="{"
		while (c:=SubStr(_s, ++n, 1)) !== ""
			InStr(" `t`n`r", c) || (
				InStr(z, c) || E(c, n), a:=(b:=s.1).__Arr, u:=0
				, InStr("}]", c) ? ((s.1==root) && E(c, n), s.RemoveAt(1), z:=s.1.__Arr ? ",]" : ",}")
				: InStr(",:", c) ? (z:=(kf:=!a && c==",") ? q : x)
				: InStr("{[", c) ? ((c=="{") ? (kf:=1, v:={}, z:=q "}") 
					: (v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v)
				: ((c==q) ? (v:=SubStr(_s, n+1, InStr(_s,q,, n+1)-n-1)
						, n+=StrLen(v)+1, (kf) && (k:=v, z:=":", u:=1)) ;string literals
					: (v:=SubStr(_s, n, min(InStr(_s,"]",,n)||M,InStr(_s,"}",,n)||M,InStr(_s,",",,n)||M,InStr(_s," ",,n)||M)-n)
						, n+=StrLen(v)-1, (v is "Number") ? v+=0:E(c, n)) ;number
				 ,(u) || (a ? k:=b.Push(v) : b[k]:=v, z:=a ? ",]" : ",}"))
			)
		return (s.1==root || E(c, n), b) ;s.Count()!=1 ;unpaired {}, []
		E(_c,_n)=>this.fthrow(Exception("Unexpected char <" _c "> in pos " _n, -2))
	}
	
	static Compact:=0, Space:=4, ErrObjType:=0, ErrObj:=0 		
	set(_o, _d:="") { ;_d:current indent
		static q:= Chr(34), ind, space
		if !IsObject(_o) ;"string" | number
			return this.ErrObj && this.fthrow(Exception("Not an object.", -1))
		(_d || space == this.Space) || ind := indent(space := this.Space)
		,(this.Compact) ? n:=d:="" : n:="`n", d:=_d ind
		,VarSetCapacity(s,1024*8)
		try a:=_o.__Arr ;trying to get a unique key
		catch ;"Unknown property" exception means its a built-in inenumerable object
			return (
				t:=Type(_o) " Object"
				,(this.ErrObjType) && this.fthrow(Exception(t " type is not supported.", -1))
				,"{" q t q "}"
			)
		for i in _o	;due to the use of a for-loop, arrays such as '[1,,3]' are detected as objects({})
			if !(a:=i=A_Index) ;a=0:associative or sparse array ([1,,3])
				break
		for k, v in _o ;recursive ;JSON standard: keys are always "strings", values r: number|"string"|{object}|[array]
			s .=  d (a ? "" : q k q ":") (isObject(v) ? this.set(v, d)
				  : (Type(v)=="String") ? q v q
				  : (Type(v)=="Float") ? format("{:g}", v) : v) "," n
		return (a ? "[" : "{") (s ? (n RTrim(s, "," n) n _d) : "") (a ? "]" : "}") ;wrap to brackets
		indent(n) => n ? indent(--n) " " : ""
	}
	
	fthrow(e) {
		throw e
	}
}

; Test runs
j:='{"r1":{"r2":{"type1":11,"type2":11},"misc":12}, "type":"ask","price":104.2,"amount":101, "a":[111,"aa"]}'

t:=A_TickCount
loop(20000)
	b:=json.get(j)
a1:=A_TickCount-t

t:=A_TickCount
loop(20000)
	s:=json.set(b)
a2:=A_TickCount-t
	
msgbox(clipboard:=a1 "|" a2 "`n`n" s)

[vvhitevvizard]

where M:=0x7FFFFFFF is declared as static inside get

u can replace it with m:=2**31-1
but why 32 bits? x64 systems use 64 bit integer. 2**63-1. nvm 32 bits is enuf for ur task - it works )
With ur talent of replacing conditional expressions to unconditional ones u could be a great Streaming SIMD Extensions coder.

Ive tested it. 930 vs 950. +2.1%. +19.8% overall. and greatly reduced code that fits on 1 screen

but ur code would fail on \t or EOL symbols. any "beautified" json input would crash. and if u add more comparisons it would become slower than regex

actually our own beautified output would crash json.get. Try that sample as input data (AHK v2 only syntax):

Code: Select all

	j:="
(
{
    "a":[
        111,
        "aa"
    ],
    "amount":101,
    "price":104.2,
    "r1":{
        "misc":12,
        "r2":{
            "type1":11,
            "type2":11
        }
    },
    "type":"ask"
}

)"