jeeswg's benchmark tests

Put simple Tips and Tricks that are not entire Tutorials in this forum
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

19 Jan 2018, 14:58

Thank you - this is the result of one of me wanting to find out some of Helgefs remarks.
I could always help you splitting the topics if you need that. Same goes for editing the first post if you want to add new stuff.
Recommends AHK Studio
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

19 Jan 2018, 15:29

- The most practical solution, I think, everything considered, is this:
- We move this thread to a different forum, Scripts and Functions perhaps.
- It's too early to try to make this a 'tidy' thread, the posts will be being changed and added to for some time to come.
- At some future date, I could think about consolidating them in one place, in a perfected form.
- I will start a new tutorial, soon, explicitly regarding how to speed up scripts. It would be a one-post wonder, that I would continue updating from time to time. It would summarise everything that I'd discovered from doing benchmark tests.
- To maintain some order in this thread, the OP will list/link to each benchmark test.
Helgef
Posts: 2990
Joined: 17 Jul 2016, 01:02
Contact:

Re: jeeswg's benchmark tests

20 Jan 2018, 14:16

Your test is well formatted nnnik :thumbup:, note that it is ms not µs and the

Code: [Select all] [Download] (Untitled.txt)GeSHi © Codebox Plus

String from a variable, variable from an object - function with byRef:

line is used twice, the last time appears to be a mistake.

Cheers.
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

20 Jan 2018, 14:40

STRINGS: SUBSTR: CROP FIRST CHARACTER V. CROP LAST CHARACTER

- SubStr(vText, 2) v. SubStr(vText, 1, -1)
In the example below, cropping the last character was roughly 7 times faster.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- I remembered seeing a nice QPC function when reading through threads, turns out it wasn't this thread though. A function by Helgef based on a function by wolf_II.
[QPC function, returns milliseconds]
Anagrams - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=19&t=34240&p=158464#p158464
[QPC function, returns seconds]
Code Puzzle Thread - Page 3 - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=17&t=25683&p=186646#p186646
- I have added my version of the function to 'ASSUME ALWAYS' in the OP. It returns milliseconds. It's basically the same as Helgef's.
Last edited by jeeswg on 01 Feb 2018, 10:34, edited 1 time in total.
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

01 Feb 2018, 00:35

Here are 7 benchmark tests. Obvious mistakes/having potentially already done these tests earlier in the thread, are concerns.

[classic dichotomy]
benchmark tests - <= v. <
< slightly faster

[curious]
benchmark tests - || v. else
|| slightly faster

[classic dichotomy]
benchmark tests - for loop v. loop (on array)
loop slightly faster

[classic dichotomy]
benchmark tests - assign key, don't care about value (value as string/number)
'0'/'1'/(blank) roughly the same, 'a' takes about 40% longer

[curious][expected to take longer, but not this much longer]
benchmark tests - get file paths (A_LoopFileFullPath v. A_LoopFileLongPath)
A_LoopFileLongPath took around 65 times longer

[classic AHK knowledge]
benchmark tests - read lines (FileRead / FileReadLine / A_LoopReadLine / .ReadLine())
FileRead fastest
A_LoopReadLine slightly slower
.ReadLine() around 5 times slower than FileRead
FileReadLine around 350 times slower than .ReadLine() (so around 1750 times slower than FileRead)

[classic AHK knowledge]
benchmark tests - append text (with/without prior VarSetCapacity)
if don't prepare capacity, around 8 times slower (this is very much dependent on how many times expansion/copying is necessary)

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Last edited by jeeswg on 01 Feb 2018, 10:35, edited 1 time in total.
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

01 Feb 2018, 03:50

From my tests the 1st test is always slightly slower than the following tests.
Recommends AHK Studio
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

01 Feb 2018, 04:11

After switching the order of some tests I got opposite results see:
< vs <=
( BTW both < and <= should be just as fast as the other since they are both a single CPU instruction )

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

Recommends AHK Studio
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

01 Feb 2018, 04:53

Thanks nnnik, I had been concerned about whether order mattered. Does anyone have any ideas?
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

01 Feb 2018, 05:06

Rather than leaving it to writing manual tests all the time we should probably write a benchmark library - that handles the tests correctly
Recommends AHK Studio
Helgef
Posts: 2990
Joined: 17 Jul 2016, 01:02
Contact:

Re: jeeswg's benchmark tests

01 Feb 2018, 05:35

for loop v. loop (on array)
loop slightly faster

For me for-loop is faster, as expected.

Cheers.


note: this function returns seconds, the original function returns milliseconds

Note: Your function returns milliseconds.
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

01 Feb 2018, 10:27

- I had thought that the for loop might be faster. Why did we get different results (did you perform the tests twice, swapping the order, did you perform the operation more times)? Why are earlier tests faster within the same script run, should we have a delay after a hotkey is executed or a dummy test?
- I've tried to incorporate every piece of advice I've been given. So getting incorrect results at this stage is really facepalm.
- Btw how do x64/x32 compare generally?
- Also I've been wondering about 'CPU' benchmark tests, or other measures, i.e. the method that takes less of a toll on the system.
- Yes, Helgef, it's milliseconds. I've now corrected it everywhere in this thread. [@nnnik: could you edit your post to make it say 'benchmark tests (ms)', thanks.]
QueryPerformanceCounter MSDN page:
Retrieves the current value of the performance counter, which is a high resolution (<1us) time stamp that can be used for time-interval measurements.
...
A pointer to a variable that receives the current performance-counter value, in counts.
QueryPerformanceFrequency MSDN page:
A pointer to a variable that receives the current performance-counter frequency, in counts per second.

Code: [Select all] [Download] GeSHi © Codebox Plus

;frequency = count / time
;count = frequency * time
;time = count / frequency
DllCall("kernel32\QueryPerformanceFrequency", Int64P,vFreq) ;count per sec
DllCall("kernel32\QueryPerformanceCounter", Int64P,vCount) ;count
MsgBox, % (vCount / vFreq) ;seconds
MsgBox, % (vCount / vFreq) * 1000 ;milliseconds
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

17 Feb 2018, 07:25

Here's an interesting one, object count keys: clone+delete v. for loop v. delete. See results at the bottom. Cheers.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

24 Feb 2018, 07:00

- I'd be interested in any benchmark tests re. optimising DllCall results, e.g. specifying dll name or not, specifying .dll or not, specifying W/A or not, using Int or "Int" and anything else. I haven't been able to get any clear-cut results so far.
- The documentation does mention about using LoadLibrary and using a function address (for dlls that aren't pre-loaded).
DllCall
https://autohotkey.com/docs/commands/DllCall.htm#load
- Also, any good example dlls/functions for testing would be helpful, it's hard to find the best ones for testing.

- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

24 Feb 2018, 07:35

jeeswg wrote:- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.

Thats due to the way modern CPUs work - the thing you are looking for is called Branch Prediction.

Also after testing it myself it seems that using a pointer isn't faster than using the dll functions name.
Recommends AHK Studio
Helgef
Posts: 2990
Joined: 17 Jul 2016, 01:02
Contact:

Re: jeeswg's benchmark tests

25 Feb 2018, 14:31

Use, quotes for the types to avoid creating unnecessary variables. #NoEnv is important when omitting quotes for types, as documented. Also, if dllcall performance is an issue, AHK_H has dynacall, which is generally faster if I am not mistaken.

@nnnik, I think script code is a little too high level for branch prediction have this effect.

Cheers.
User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

25 Feb 2018, 15:29

@Helgef yes it is. However AutoHotkey itself will speed up after the first test. ( The low level code that handles all sort of stuff )
Thats consistent with our findings that the 1st test is the slower than the second test of the same type no matter what happens.
Depending on which things we do our branch prediction might already be trained for specific actions in autohotkey while it isn't trained for others.
Recommends AHK Studio
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

13 Mar 2018, 02:07

Why do some people use the MulDiv function, instead of built-in operators?

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus

User avatar
nnnik
Posts: 2908
Joined: 30 Sep 2013, 01:01
Location: Germany

Re: jeeswg's benchmark tests

13 Mar 2018, 02:34

Have you compared the precision of the results?
That would be my only guess
Recommends AHK Studio
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

02 Jun 2018, 16:23

- I ran some tests against A_ variables:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- Here are two batches of results for the slowest variables totalling 1 second or more for 1000 reads:

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- Summary of results:
very slow: A_IPAddress1/2/3/4
quite slow: A_IsAdmin, A_UserName
also slow: A_CaretX/Y, A_ComputerName, A_Cursor, A_Now/A_NowUTC
also slow: dir-related variables:
A_AppData/A_AppDataCommon
A_Desktop/A_DesktopCommon
A_MyDocuments
A_ProgramFiles
A_Programs/A_ProgramsCommon
A_StartMenu/A_StartMenuCommon
A_Startup/A_StartupCommon
A_Temp

- Note: All of the variables mentioned in WINDOWS FOLDER LOCATIONS, here:
jeeswg's Explorer tutorial - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=7&t=31755
appeared in the list of slow variables, apart from A_ComSpec and A_WinDir.
- Note: Clearly tests of file/registry loop variables should be done within loops for more useful results.

- Threads that prompted the tests:
A_LoopFileLongPath MUCH slower than A_LoopFileFullPath - Issues - AutoHotkey Community
https://autohotkey.com/board/topic/55613-a-loopfilelongpath-much-slower-than-a-loopfilefullpath/
Possible A_LoopFileLongPath bug - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=42891
A_IPAddress Performance - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=14&t=49910
- Note also:
Test build - Obj.Count(), OnError(), long paths, experimental switch-case - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=24&t=47682
Fixes a significant performance issue with A_LoopFileLongPath (and A_LoopFileShortPath is also optimized).
User avatar
jeeswg
Posts: 4508
Joined: 19 Dec 2016, 01:58
Location: UK

Re: jeeswg's benchmark tests

02 Jun 2018, 20:53

- Here's a mod for the script above, you can include/comment out specifics line to give you results within/outside of a file/registry loop.

Code: [Select all] [Expand] [Download] GeSHi © Codebox Plus


- I came across 5 A_LoopFileXXX variables that took more than 1 second for 1000 reads in the file loop.
- I found no A_LoopRegXXX variables that took more than 1 second for 1000 reads in the registry loop.
- Slower variables in the file loop:

- A_LoopFileLongPath and A_LoopFileShortPath are addressed in this AHK test build:
Test build - Obj.Count(), OnError(), long paths, experimental switch-case - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=24&t=47682
- My code here essentially recreates the logic used in an AHK file loop, most of the key file data is placed into a WIN32_FIND_DATA struct, which the A_LoopFileXXX variables no doubt refer to. AHK has to convert those UTC dates to local dates, which probably explains the relative slowness.
259-char path limit workarounds - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=26170
- (I had proposed A_LoopFileTimeModifiedUTC and A_LoopFileTimeCreatedUTC variables (and A_LoopFileTimeAccessedUTC for completeness), as they would be faster, but also more useful, when comparing files across DST and time zone differences.)
- (Similarly, an A_LoopFileAttribValue variable would be faster than A_LoopFileAttrib, as you'd get the raw number from the struct instead of converting the number to letters (from the list 'RASHNDOCT'), and more useful, as any unusual properties would also be listed.)

Return to “Tips and Tricks”

Who is online

Users browsing this forum: No registered users and 1 guest