AHK v2: converting/optimizing scripts Topic is solved

[vvhitevvizard]

btw I'm sorry for my bad habit of re-renaming vars. Those 1-letter names r not random actually. I just used to some local vars naming convention. i, k, v for index, key, value (and not only inside for-loops), n for size or length, x,y,w,h for coords and width/height, s for strings (especially for output), m for arrays returned by RegexMatch/StrSplit, t for timestamp or temporary array, c for uchar, g for Gui|GuiControl object, et cetera et cetera. So those little literals r actually self-explaining for me

[oif2003]

The things we do for speed/readability! I haven't noticed a decrease in speed from all the spaces and parenthesis I added. I think loop parse takes off a modest 5-10% but only if you replace c with A_LoopField. We might be reaching the point of diminishing returns; there might not be much more fat left to trim.

[vvhitevvizard]

I haven't noticed a decrease in speed from all the spaces and parenthesis I added.

I just remove white space here and there, e.g. between assignments := so that more code fits into my screen. Readability is NOT an issue with syntax highlighting. In my 200Kb quick "Notepad2" editor it looks like (with operators emphasized with the most bright color, somewhat buggy tho; e.g. . is considered as literal):


2.
I haven't tested ur Loop Parse yet. Im slow =) But from my prev. tests of Loop Parse vs Instr()|SubStr() for AHK 1.1, the latter 2 seemed faster. And I hate ugly command-style of Loop Parse in AHK v2. I recall there was AHK alpha-version with LoopParse (w.o space between) but lexikos didn't like this and removed it back.
Out of curiosity, can we arrange a quick synthetic benchmark to compare Loop Parse vs its more optimized counterparts?

We might be reaching the point of diminishing returns; there might not be much more fat left to trim.

Mark my words: there is still room for speed improvement by the cost of negligible script size's increase.

[oif2003]

Loop parse beats while loop by a lot in my simple benchmark. But that's before you load it down with actual work. I didn't think about replacing other bits with loop parse. I tested InStr("ab", x) vs x="a"||x="b" and I remember InStr being faster, so I left them alone. I merged all the single lines, and couldn't find a speed difference... 124 lines vs 34 lines both run at the same speed.
As a side effect, I now I have a 750 character long loop parse line

Spoiler

Code: Select all

loop parse _s
			(A_Index < n) || InStr(" `t`n`r", A_LoopField) || (n := A_Index, InStr(z, A_LoopField) || E(A_LoopField, n), u:=0, InStr("}]", A_LoopField) ? ((s.1=root) && E(A_LoopField, n), s.RemoveAt(1), z := (a:=(b:=s.1).__Arr) ? ",]" : ",}"): InStr(",:", A_LoopField) ?(z:=(kf:=!a && A_LoopField = ",") ? q : x): InStr("{[", A_LoopField) ? ((A_LoopField = "{") ? (kf:=1, v:={}, z:=q "}"):(v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v, a:=(b:=s.1).__Arr):((A_LoopField = q) ?(_n:=InStr(_s,q,, n+1),v:=SubStr(_s, n+1, _n-n-1),n:=_n,(kf) && (k:=v, z:=":", u:=1)):(_n:=SubStr(_s, n) ~= "[\]\},\s]|$",v:=SubStr(_s, n, _n-1),n+=_n-2, (v is "Number") ? v+=0 :E(A_LoopField, n)), (u) || ((a) ? (k:=b.Push(v), z:=",]") : (b[k] := v, z := ",}"))), n++)

Maybe if you rewrite json.set without recursion it will run faster, but most of the time is spent in json.get anyway. You can also explore mcode... but that's like cheating

[vvhitevvizard]

I think loop parse takes off a modest 5-10% but only if you replace c with A_LoopField.

We could just assign c:=A_LoopField once in the top of the cycle and be done with it. the code's compactness stays intact =)

[oif2003]

I think loop parse takes off a modest 5-10% but only if you replace c with A_LoopField.

We could just assign c:=A_LoopField once in the top of the cycle and be done with it. =)

Try it, it didn't work for me, as that was the first thing I did
Seems like it must be updated on every iteration where A_Index>=n, which is a lot, so maybe that's why. Perhaps by using A_LoopField the interpreter knows it is a constant? I don't know. I have little idea of the machinery underneath.
But then the same trick didn't work for A_Index when I tried to replace most of the n's

[vvhitevvizard]

Loop parse beats while loop by a lot in my simple benchmark. But that's before you load it down with actual work.

Lets just create benchmark with real-world use case.

I tested InStr("ab", x) vs x="a"||x="b" and I remember InStr being faster, so I left them alone.

Thats b/c the first is a single pseudo-code command and the latter consists of 3 pseudo-code subcommands (2 var== comparisons and || operator). The fun begins when u compare with long-length line of tokens. First variant should be MUCH faster this way cuz the comparison is done within native compiled code of the built-in function instead of interpreting pseudo-code iterations. Correct me if I'm wrong.

I merged all the single lines, and couldn't find a speed difference... 124 lines vs 34 lines both run at the same speed.
As a side effect, I now I have a 750 character long loop parse line

what the hell is that Gimme some time to test it

Maybe if you rewrite json.set without recursion it will run faster, but most of the time is spent in json.get anyway. You can also explore mcode... but that's like cheating

Disposing with recursiveness would impose push/pop functionality or arrays to be added. I guess AHK handles its own inner function's stack (func addresses + args + local vars + closures etc etc) in a better and more optimized way. I recall recursiveness was always bad for low-level languages like assembler or plain C and I tried to avoid using it by all means. With an interpreter like AHK it might be a different story. we should test it.
m-coding time-critical parts in AHK scripts is a must. And ability of placing super-optimized machine code (including extended SIMD instructions) thats what makes AHK special compared to other script languages. The only concern here is not to lose completely 2 important things: algorithms readability and ability to be edited ad-hoc. Thats why I offered to m-code only small parts of the json routines.

[oif2003]

The only concern here is not to lose completely 2 important things: algorithms readability and ability to be edited ad-hoc.

What? You mean to tell me my json.get is neither readable nor flexible?!

Spoiler

Code: Select all

	Get(ByRef _s){ 
		static q:=Chr(34), p:={"__Arr":1}, x:=q "{[0123456789-"
		n:=0, k:="", kf:=0, s:=[b:=root:={}], z:="{", E(_c,_n)=>this._T("Unexpected char <" _c "> in pos " _n)
		loop parse _s
			(A_Index < n) || InStr(" `t`n`r", A_LoopField) || (n := A_Index, InStr(z, A_LoopField) || E(A_LoopField, n), u:=0, InStr("}]", A_LoopField) ? ((s.1=root) && E(A_LoopField, n), s.RemoveAt(1), z := (a:=(b:=s.1).__Arr) ? ",]" : ",}"): InStr(",:", A_LoopField) ?(z:=(kf:=!a && A_LoopField = ",") ? q : x): InStr("{[", A_LoopField) ? ((A_LoopField = "{") ? (kf:=1, v:={}, z:=q "}"):(v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v, a:=(b:=s.1).__Arr):((A_LoopField = q) ?(_n:=InStr(_s,q,, n+1),v:=SubStr(_s, n+1, _n-n-1),n:=_n,(kf) && (k:=v, z:=":", u:=1)):(_n:=SubStr(_s, n) ~= "[\]\},\s]|$",v:=SubStr(_s, n, _n-1),n+=_n-2, (v is "Number") ? v+=0 :E(A_LoopField, n)), (u) || ((a) ? (k:=b.Push(v), z:=",]") : (b[k] := v, z := ",}"))), n++)
		return ((s.1=root) || E(_, n), s[1,""])
	}

[vvhitevvizard]

About m-coding. Honestly, I havent installed C/C++ builder environments for like 7 years.
We have to start from converting this sub-line SubStr(_s, n, (SubStr(_s, n) ~= "[\]\},\s]|$")-1 into machine code.
2 ALL: regex code logic (pseudocode) here ~= "[\]\},\s]|$ is as follows:

Code: Select all

if char from string == "]" or "}" or "," or " " or "\t" or "\n" or "r"
    then return 1
return 0

Can someone help us out and compile it? For x64. I need an intermediate .obj file

[vvhitevvizard]

What? You mean to tell me my json.get is neither readable nor flexible?!

Flexibility (ability to edit) reduces when u inline more and more stuff and add more and more nested parentheses. Inlining is obviously good but to some extent.

[vvhitevvizard]

What? You mean to tell me my json.get is neither readable nor flexible?!

I benched it. It gets worse CPUtime-wise. 937 vs 844
And the code becomes increasingly more difficult to edit. I have some ideas of optimizing its logic still.

btw, it actually scores 959 with 1 line looper (ur code intact) and scores 937 (better) when I split the huge loop into 10 lines. Looks like it overloads AHK interpreter as well.
Ur brilliant mind obviously moves in the wrong direction today.

[oif2003]

What? You mean to tell me my json.get is neither readable nor flexible?!

I benched it. It gets worse CPUtime-wise. 937 vs 844
And the code becomes increasingly more difficult to edit. I have some ideas of optimizing its logic still.

Actually it scores 955 with 1 line looper (ur code intact) and scores 937 (better) when I split the huge loop into 10 lines. Ur brilliant mind obviously moves in the wrong direction today.

That's strange. Which version were you benching against? I am getting better results than https://www.autohotkey.com/boards/viewt ... 80#p252861
also I bench using this:

Code: Select all

	j:="
(
{
    "a":[
        111,
        "aa"
    ],
    "amount":101,
    "price":104.2,
    "r1":{
        "misc":12,
        "r2":{
            "type1":11,
            "type2":11
        }
    },
    "type":"ask"
}

)"

[vvhitevvizard]

Code: Select all

That's strange. Which version were you benching against? I am getting better results than ...

Yeah, against this ver. with the same test string j:='{"r1":{"r2":{"type1":11,"type2":11},"misc":12}, "type":"ask","price":104.2,"amount":101, "a":[111,"aa"]}'


I tested like 30 times. Not a single time it got even close. It sits @ 12.5% higher CPUtime overall.
best was 938. worst was over 1000


At any rate we haven't done modifying the algo to finalize it (wrap all stuff in single line) like this

[oif2003]

Hey, it's not the same string if u remove all the space, at least my tests give different results.

[vvhitevvizard]

Hey, it's not the same string if u remove all the space, at least my tests give different results.

absolutely same string (j:=). w/o any whitespace added except 2 " " spaces for the test sake. I never modified that string.
settings r the same as well:
static Compact:=0, Space:=4

best timings of both is: 937 vs 843

[oif2003]

Same result for me. Try feeding both the same json strings with newlines and tabs. This is weird. Loop parse should be faster, too. As to how much, I do not know, but tested independently loop parse was several times faster than while loop.

[vvhitevvizard]

thats interesting indeed.

with beautified test string

Code: Select all

j:="
(
{
    "a":[
        111,
        "aa"
    ],
    "amount":101,
    "price":104.2,
    "r1":{
        "misc":12,
        "r2":{
            "type1":11,
            "type2":11
        }
    },
    "type":"ask"
}

)"

ur 1-line looper scores 1234 and v1.1 scores 1375. +10% performance improvement here and degradation for whitespace free json.
But the majority of json data circulating is optimized for size (in compact mode, no padding left). So I concede ur last ver improves conditionally but we move in wrong direction overall.

[oif2003]

Ain't that weird... when benching json string without tabs and newlines, 1.1 is faster. It's the other way around when I feed them json with tabs and newlines. Oh well, so much for loop parse then. On the bright side, one can always add a few more lines of code. Loop parse takes only 2 lol

My C has failed me. I think when I call my C dll it wants to convert strings to AStr form. It's super slow.

[vvhitevvizard]

So suboptimal part of v1.1 is here InStr(" `t`n`r", c) || ( with parsing whitespace

My C has failed me. I think when I call my C dll it wants to convert strings to AStr form. It's super slow.

Tell the compiler that's a buffer* of chars (well, 2-byte DWORD Unicode tokens actually). Not a zero terminated string. Hell, it gets complicated already: we have to use Unicode assumption as default for optimizations and stick to it with m-coding. well, I use only Unicode x64 version of AHK v2 anyways and never ever resorted to others.


btw, here is c:=A_LoopField to reduce code size. It works. I just assign c as the first instruction within loop. But performance drops by 9% for the beautified string and by 4.5% for the compact string for some reason. So it just continues the course of Loop Parse weirdness. =)

Spoiler

Code: Select all

	Get(ByRef _s){ 
		static q:=Chr(34), p:={"__Arr":1}, x:=q "{[0123456789-"
		n:=0, k:="", kf:=0, s:=[b:=root:={}], z:="{", E(_c,_n)=>this._T("Unexpected char <" _c "> in pos " _n)
		loop parse _s
			c:=A_LoopField, (A_Index < n) || InStr(" `t`n`r", c) || (n := A_Index, InStr(z, c) || E(c, n), u:=0, InStr("}]", c) ? ((s.1=root) && E(c, n), s.RemoveAt(1), z := (a:=(b:=s.1).__Arr) ? ",]" : ",}"): InStr(",:", c) ?(z:=(kf:=!a && c = ",") ? q : x): InStr("{[", c) ? ((c = "{") ? (kf:=1, v:={}, z:=q "}"):(v:=new p, z:=x "]"), s.InsertAt(1,v), a ? k:=b.Push(v) : b[k]:=v, a:=(b:=s.1).__Arr):((c = q) ?(_n:=InStr(_s,q,, n+1),v:=SubStr(_s, n+1, _n-n-1),n:=_n,(kf) && (k:=v, z:=":", u:=1)):(_n:=SubStr(_s, n) ~= "[\]\},\s]|$",v:=SubStr(_s, n, _n-1),n+=_n-2, (v is "Number") ? v+=0 :E(c, n)), (u) || ((a) ? (k:=b.Push(v), z:=",]") : (b[k] := v, z := ",}"))), n++)
		return ((s.1=root) || E(_, n), s[1,""])
	}

[vvhitevvizard]

it looks like A_Index, A_LoopField, etc r treated in a more optimized way unlike usual user's vars.
I guess thats the reason why the following is true:

Loop parse beats while loop by a lot in my simple benchmark.