jeeswg's benchmark tests

[nnnik]

Thank you - this is the result of one of me wanting to find out some of Helgefs remarks.
I could always help you splitting the topics if you need that. Same goes for editing the first post if you want to add new stuff.

[jeeswg]

- The most practical solution, I think, everything considered, is this:
- We move this thread to a different forum, Scripts and Functions perhaps.
- It's too early to try to make this a 'tidy' thread, the posts will be being changed and added to for some time to come.
- At some future date, I could think about consolidating them in one place, in a perfected form.
- I will start a new tutorial, soon, explicitly regarding how to speed up scripts. It would be a one-post wonder, that I would continue updating from time to time. It would summarise everything that I'd discovered from doing benchmark tests.
- To maintain some order in this thread, the OP will list/link to each benchmark test.

[Helgef]

Your test is well formatted nnnik , note that it is ms not µs and the

Code: [Select all] [Download] (Untitled.txt)GeSHi © Codebox Plus

String from a variable, variable from an object - function with byRef:

line is used twice, the last time appears to be a mistake.

Cheers.

[jeeswg]

STRINGS: SUBSTR: CROP FIRST CHARACTER V. CROP LAST CHARACTER

- SubStr(vText, 2) v. SubStr(vText, 1, -1)
In the example below, cropping the last character was roughly 7 times faster.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

q:: ;crop first character v. crop last character
oQPC := []
vNum := 10000
vLen := 100000
VarSetCapacity(vText, vLen*2)
Loop, % vLen
	vText .= "a"
vText2 := SubStr(vText, 2)

vQPC := QPC()
Loop, % vNum
	vText2 := SubStr(vText, 2)
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	vText2 := SubStr(vText, 1, -1)
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oQPC := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;crop first	crop last
;1126.564967	150.033698
;1086.523708	146.570629
;1133.453045	170.595727
;1170.836918	173.002940
;1096.475436	151.476573

- I remembered seeing a nice QPC function when reading through threads, turns out it wasn't this thread though. A function by Helgef based on a function by wolf_II.
[QPC function, returns milliseconds]
Anagrams - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=19&t=34240&p=158464#p158464
[QPC function, returns seconds]
Code Puzzle Thread - Page 3 - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=17&t=25683&p=186646#p186646
- I have added my version of the function to 'ASSUME ALWAYS' in the OP. It returns milliseconds. It's basically the same as Helgef's.

[jeeswg]

Here are 7 benchmark tests. Obvious mistakes/having potentially already done these tests earlier in the thread, are concerns.

[classic dichotomy]
benchmark tests - <= v. <
< slightly faster

[curious]
benchmark tests - || v. else
|| slightly faster

[classic dichotomy]
benchmark tests - for loop v. loop (on array)
loop slightly faster

[classic dichotomy]
benchmark tests - assign key, don't care about value (value as string/number)
'0'/'1'/(blank) roughly the same, 'a' takes about 40% longer

[curious][expected to take longer, but not this much longer]
benchmark tests - get file paths (A_LoopFileFullPath v. A_LoopFileLongPath)
A_LoopFileLongPath took around 65 times longer

[classic AHK knowledge]
benchmark tests - read lines (FileRead / FileReadLine / A_LoopReadLine / .ReadLine())
FileRead fastest
A_LoopReadLine slightly slower
.ReadLine() around 5 times slower than FileRead
FileReadLine around 350 times slower than .ReadLine() (so around 1750 times slower than FileRead)

[classic AHK knowledge]
benchmark tests - append text (with/without prior VarSetCapacity)
if don't prepare capacity, around 8 times slower (this is very much dependent on how many times expansion/copying is necessary)

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

return

;==================================================

;q:: ;benchmark tests - <= v. <
oQPC := []
vNum := 1000000

vQPC := QPC()
Loop, % vNum
	var := 2 <= 3
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	var := 2 < 3
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oQPC := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;3 <= 3		3 < 3
;174.843555	161.943055
;184.344235	157.053729
;179.711071	159.100035
;186.744636	157.246171
;190.099553	161.883184

;benchmark tests (ms)
;2 <= 3		2 < 3
;173.132954	166.935873
;184.478516	154.672572
;193.881265	155.923448
;175.886167	156.159512
;183.458571	155.182758

;==================================================

;q:: ;benchmark tests - || v. else
oQPC := []
vNum := 1000000
a := 0, b := 0, c := 1

vQPC := QPC()
Loop, % vNum
	if a || b || c
	{
	}
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	if a
	{
	}
	else if b
	{
	}
	else if c
	{
	}
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oQPC := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;||		else
;188.241840	221.797423
;192.765525	216.007465
;186.399949	216.933756
;189.934480	211.650564
;187.081625	213.128950

;==================================================

;q:: ;benchmark tests - for loop v. loop (on array)
oQPC := []
oArray := StrSplit("abcdefghijklmnopqrstuvwxyz")
vNum := 100000

vQPC := QPC()
Loop, % vNum
	for _, vValue in oArray
		vValue2 := vValue
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	Loop, % oArray.Length()
		vValue2 := oArray[A_Index]
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oArray := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;for loop	loop
;655.221461	559.977317
;607.630822	571.460156
;583.126885	537.965727
;616.441273	542.546290
;598.056586	537.650549

;==================================================

;q:: ;benchmark tests - assign key, don't care about value (value as string/number)
oQPC := []
vNum := 1000000
vNum2 := 10

oArray := []
vQPC := QPC()
Loop, % vNum
{
	oArray := []
	Loop, % vNum2
		oArray[A_Index] := 0
}
oQPC.Push(QPC()-vQPC)

oArray := []
vQPC := QPC()
Loop, % vNum
{
	oArray := []
	Loop, % vNum2
		oArray[A_Index] := 1
}
oQPC.Push(QPC()-vQPC)

oArray := []
vQPC := QPC()
Loop, % vNum
{
	oArray := []
	Loop, % vNum2
		oArray[A_Index] := ""
}
oQPC.Push(QPC()-vQPC)

oArray := []
vQPC := QPC()
Loop, % vNum
{
	oArray := []
	Loop, % vNum2
		oArray[A_Index] := "a"
}
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2 "`t" oQPC.3 "`t" oQPC.4
oArray := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;CATEG 1	CATEG 2
;2397.100274	2219.101515	2212.104300	3065.423231
;2228.236554	2188.672056	2203.594914	3018.515121
;2246.667000	2281.473461	2359.130911	3021.296987
;2259.591449	2240.394226	2218.417702	3008.272468
;2248.675674	2225.712134	2200.738638	3009.223134

;==================================================

;q:: ;benchmark tests - get file paths (A_LoopFileFullPath v. A_LoopFileLongPath)
oQPC := []
vNum := 10
vDir1 := A_ScriptDir

vQPC := QPC()
Loop, % vNum
	Loop, Files, % vDir1 "\*", F
		vPath := A_LoopFileFullPath
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	Loop, Files, % vDir1 "\*", F
		vPath := A_LoopFileLongPath
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oArray := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;CATEG 1	CATEG 2
;11.150982	1521.535401
;21.380378	1428.592872
;19.755307	1403.026225
;21.218299	1407.470367
;24.869150	1406.404663

;==================================================

;q:: ;benchmark tests - read lines (FileRead / FileReadLine / A_LoopReadLine / .ReadLine())
;note: creates file: A_Desktop "\z benchmark test num list.txt"
oQPC := []
vDir1 := A_ScriptDir
vNum := 1000

vPath := A_Desktop "\z benchmark test num list.txt"
if !FileExist(vPath)
{
	VarSetCapacity(vOutput, vLen*(StrLen(vLen)+2)*2)
	vLen := 10000
	Loop, % vLen
		vOutput .= A_Index "`r`n"
	vOutput := SubStr(vOutput, 1, -2)
	FileAppend, % vOutput, % "*" vPath
}
FileRead, vText, % vPath
StrReplace(vText, "`n", "", vLen), vLen += 1
vText := ""

vQPC := QPC()
Loop, % vNum
{
	FileRead, vText, % vPath
	Loop, Parse, vText, `n, `r
		vTemp := A_LoopField
}
oQPC.Push(QPC()-vQPC)

if 0
{
	vQPC := QPC()
	Loop, % vNum
	{
		Loop, % vLen
			FileReadLine, vTemp, % vPath, % A_Index
	}
	oQPC.Push(QPC()-vQPC)
}
else
	oQPC.Push("")

vQPC := QPC()
Loop, % vNum
{
	Loop, Read, % vPath
		vTemp := A_LoopReadLine
}
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
{
	oFile := FileOpen(vPath, "r")
	Loop, % vLen
		vTemp := oFile.ReadLine()
	oFile.Close()
}
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2 "`t" oQPC.3 "`t" oQPC.4
oArray := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;vNum := 1
;FileRead	FileReadLine	A_LoopReadLine	.ReadLine()
;1.427069	2302.362469	1.400128	6.431433
;4.239726	2197.148941	1.378317	6.136354
;4.433451	2200.732650	1.549377	6.937344
;1.572470	2270.846778	1.380027	6.105563
;4.382133	2175.230578	1.375751	6.162014

;benchmark tests (ms)
;vNum := 100
;FileRead	FileReadLine	A_LoopReadLine	.ReadLine()
;1355.364595			1463.975378	6748.853790
;1315.788550			1423.718086	6300.894773
;1286.870408			1393.392546	6417.610384
;1345.471760			1450.831972	6503.906372
;1270.004734			1405.458700	6337.372063

;==================================================

;q:: ;benchmark tests - append text (with/without prior VarSetCapacity)
oQPC := []
vNum := 1000000
vNum2 := 1

vQPC := QPC()
Loop, % vNum2
{
	vOutput := ""
	VarSetCapacity(vOutput, vNum*StrLen(vNum)*2)
	Loop, % vNum
		vOutput .= A_Index
}
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum2
{
	vOutput := ""
	VarSetCapacity(vOutput, 0)
	Loop, % vNum
		vOutput .= A_Index
}
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oArray := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;prepare	don't prepare
;234.838192	1797.717545
;240.902701	1824.915678
;229.700828	1807.288787
;250.686058	1815.534740
;213.140070	1812.087451

;==================================================

[nnnik]

From my tests the 1st test is always slightly slower than the following tests.

[nnnik]

After switching the order of some tests I got opposite results see:
< vs <=
( BTW both < and <= should be just as fast as the other since they are both a single CPU instruction )

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

SetBatchLines, -1
t::
Thread, Priority, 2147483647
Process, Priority,, High
oQPC := []
vNum := 1000000

vQPC := QPC()
Loop, % vNum
	var := 2 < 3
oQPC.Push(QPC()-vQPC)

vQPC := QPC()
Loop, % vNum
	var := 2 <= 3
oQPC.Push(QPC()-vQPC)

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (seconds)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oQPC := ""
Process, Priority,, Normal
MsgBox, % Clipboard
return


QPC() {
	static frequency := "", init := DllCall( "QueryPerformanceFrequency", "UInt64*", frequency )
	DllCall( "QueryPerformanceCounter", "UInt64*", time )
	return time * 1000 / frequency
}

[jeeswg]

Thanks nnnik, I had been concerned about whether order mattered. Does anyone have any ideas?

[nnnik]

Rather than leaving it to writing manual tests all the time we should probably write a benchmark library - that handles the tests correctly

[Helgef]

for loop v. loop (on array)
loop slightly faster

For me for-loop is faster, as expected.

Cheers.

---

note: this function returns seconds, the original function returns milliseconds

Note: Your function returns milliseconds.

[jeeswg]

- I had thought that the for loop might be faster. Why did we get different results (did you perform the tests twice, swapping the order, did you perform the operation more times)? Why are earlier tests faster within the same script run, should we have a delay after a hotkey is executed or a dummy test?
- I've tried to incorporate every piece of advice I've been given. So getting incorrect results at this stage is really facepalm.
- Btw how do x64/x32 compare generally?
- Also I've been wondering about 'CPU' benchmark tests, or other measures, i.e. the method that takes less of a toll on the system.
- Yes, Helgef, it's milliseconds. I've now corrected it everywhere in this thread. [@nnnik: could you edit your post to make it say 'benchmark tests (ms)', thanks.]
QueryPerformanceCounter MSDN page:
Retrieves the current value of the performance counter, which is a high resolution (<1us) time stamp that can be used for time-interval measurements.
...
A pointer to a variable that receives the current performance-counter value, in counts.
QueryPerformanceFrequency MSDN page:
A pointer to a variable that receives the current performance-counter frequency, in counts per second.

Code: [Select all] [Download] GeSHi © Codebox Plus

;frequency = count / time
;count = frequency * time
;time = count / frequency
DllCall("kernel32\QueryPerformanceFrequency", Int64P,vFreq) ;count per sec
DllCall("kernel32\QueryPerformanceCounter", Int64P,vCount) ;count
MsgBox, % (vCount / vFreq) ;seconds
MsgBox, % (vCount / vFreq) * 1000 ;milliseconds

[jeeswg]

Here's an interesting one, object count keys: clone+delete v. for loop v. delete. See results at the bottom. Cheers.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

q:: ;object count keys: clone+delete v. for loop v. delete
oQPC := []
;MsgBox, % A_BatchLines

FileRead, vText, % A_ScriptFullPath
obj := {}
vIndex := 0
Loop, 100
;Loop, 10
	Loop, Parse, vText, `n, `r
		obj["z" (vIndex++) A_LoopField] := ""

vQPC := QPC()
obj2 := obj.Clone()
vCount2 := obj2.Delete("", Chr(65535))
vCount1 := obj2.Delete(obj2.MinIndex(), obj2.MaxIndex())
oQPC.Push(QPC()-vQPC)
;MsgBox, % (vCount1 + vCount2) " " vCount1 " " vCount2

vQPC := QPC()
vCount := 0
for k in obj
	vCount++
oQPC.Push(QPC()-vQPC)
;MsgBox, % vCount

vQPC := QPC()
vCount2 := obj.Delete("", Chr(65535))
vCount1 := obj.Delete(obj.MinIndex(), obj.MaxIndex())
oQPC.Push(QPC()-vQPC)
;MsgBox, % (vCount1 + vCount2) " " vCount1 " " vCount2

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2 "`t" oQPC.3
oQPC := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;clone+delete	for loop	delete
;26.080144	30.588842	9.340699
;28.071703	31.812344	8.898938
;28.127725	31.804219	9.552812
;34.065197	31.749052	9.851739
;33.889861	32.787383	9.871410

[jeeswg]

- I'd be interested in any benchmark tests re. optimising DllCall results, e.g. specifying dll name or not, specifying .dll or not, specifying W/A or not, using Int or "Int" and anything else. I haven't been able to get any clear-cut results so far.
- The documentation does mention about using LoadLibrary and using a function address (for dlls that aren't pre-loaded).
DllCall
https://autohotkey.com/docs/commands/DllCall.htm#load
- Also, any good example dlls/functions for testing would be helpful, it's hard to find the best ones for testing.

- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.

[nnnik]

- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.

Thats due to the way modern CPUs work - the thing you are looking for is called Branch Prediction.

Also after testing it myself it seems that using a pointer isn't faster than using the dll functions name.

[Helgef]

Use, quotes for the types to avoid creating unnecessary variables. #NoEnv is important when omitting quotes for types, as documented. Also, if dllcall performance is an issue, AHK_H has dynacall, which is generally faster if I am not mistaken.

@nnnik, I think script code is a little too high level for branch prediction have this effect.

Cheers.

[nnnik]

@Helgef yes it is. However AutoHotkey itself will speed up after the first test. ( The low level code that handles all sort of stuff )
Thats consistent with our findings that the 1st test is the slower than the second test of the same type no matter what happens.
Depending on which things we do our branch prediction might already be trained for specific actions in autohotkey while it isn't trained for others.

[jeeswg]

Why do some people use the MulDiv function, instead of built-in operators?

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

q:: ;MulDiv v. built-in operators
WinGet, hWnd, ID, A
oQPC := []
vCount := 1000000
;vNum1 := 20, vNum2 := 20, vNum3 := 10
vNum1 := 5, vNum2 := 5, vNum3 := 10

vQPC := QPC()
Loop, % vCount
	vRet := Round(vNum1 * vNum2 / vNum3)
oQPC.Push(QPC()-vQPC)
;MsgBox, % vRet

vQPC := QPC()
Loop, % vCount
	vRet := DllCall("kernel32.dll\MulDiv", Int,vNum1, Int,vNum2, Int,vNum3)
oQPC.Push(QPC()-vQPC)
;MsgBox, % vRet

if !(SubStr(Clipboard, 1, 16) = ";benchmark tests")
	Clipboard := ";benchmark tests (ms)`r`n;CATEG 1`t" "CATEG 2"
Clipboard .= "`r`n;" oQPC.1 "`t" oQPC.2
oQPC := ""
MsgBox, % Clipboard
return

;benchmark tests (ms)
;operators	MulDiv
;269.937165	556.748557
;270.149708	552.227010
;320.876307	550.009216
;281.163842	535.614933
;285.386034	521.193281

[nnnik]

Have you compared the precision of the results?
That would be my only guess