jeeswg's benchmark tests

Post a reply

Confirmation code
Enter the code exactly as it appears. All letters are case insensitive.
Smilies
:D :) ;) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :!: :?: :idea: :| :mrgreen: :geek: :ugeek: :arrow: :angel: :clap: :crazy: :eh: :lolno: :problem: :shh: :shifty: :sick: :silent: :think: :thumbup: :thumbdown: :salute: :wave: :wtf: :yawn: :facepalm: :bravo: :dance: :beard: :morebeard: :xmas: :HeHe: :trollface: :cookie: :rainbow: :monkeysee: :monkeysay: :happybday: :headwall: :offtopic: :superhappy: :terms: :beer:
View more smilies

BBCode is ON
[img] is OFF
[flash] is OFF
[url] is ON
Smilies are ON

Topic review
   

Expand view Topic review: jeeswg's benchmark tests

Re: jeeswg's benchmark tests

Post by nnnik » 06 Sep 2018, 02:09

No I dont mean to ignore the first result - I'm not sure that will work.
The only way that you can be sure that it will work is by setting AHKs priority making the thread uninterupteable and reducing the amount of external influences. (e.g. a Hotkey would be an external influence)

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 06 Sep 2018, 01:39

- Point taken, but what's wrong with a hotkey?
- Do you mean ignore the first result? I already do that.
- I usually alter the tests until a difference is clear, but yes, I'll create stand-alone scripts in future. I've created a new folder just now, this will make it easier to manage and share my benchmark tests in future. (I will port old benchmark tests there also.)
- Would you not suggest to use a separate script for each version of the test?
- My intention would be to output results using stdout.
- If you were so inclined, you could create a zip with what you consider the perfect setup, with an example benchmark test. Cheers.

Re: jeeswg's benchmark tests

Post by nnnik » 06 Sep 2018, 01:32

2 things:
Don't trigger your benchmarks with a Hotkey and please let AutoHotkey do an unmeasured testrun before all serious runs.
e.g.:

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 05 Sep 2018, 17:22

- I was interested as to whether VarSetCapacity was much slower for: (a) larger capacities, and (b) using fill bytes.
- I would say that the cost is pretty low.

Code: [Select all] [Expand]GeSHi © Codebox Plus


Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 05 Sep 2018, 16:45

- I wanted to be able to check if a string was ASCII. If ASCII, save a file without a BOM, else, save a file as UTF-8 with a BOM.
- Loop can be slow, and RegEx can be slow, so it was a battle of two 'slownesses'. RegEx was much faster in these examples.

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 01 Sep 2018, 21:25

with/without obj.SetCapacity() (SetCapacity() can reduce the time needed by 37%)
I try a small number of pushes (100), multiple times

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 02 Jun 2018, 21:59

modulo v. compare and reset (compare and reset is faster)

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 02 Jun 2018, 20:53

- Here's a mod for the script above, you can include/comment out specifics line to give you results within/outside of a file/registry loop.

Code: [Select all] [Expand]GeSHi © Codebox Plus


- I came across 5 A_LoopFileXXX variables that took more than 1 second for 1000 reads in the file loop.
- I found no A_LoopRegXXX variables that took more than 1 second for 1000 reads in the registry loop.
- Slower variables in the file loop:

- A_LoopFileLongPath and A_LoopFileShortPath are addressed in this AHK test build:
Test build - Obj.Count(), OnError(), long paths, experimental switch-case - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=24&t=47682
- My code here essentially recreates the logic used in an AHK file loop, most of the key file data is placed into a WIN32_FIND_DATA struct, which the A_LoopFileXXX variables no doubt refer to. AHK has to convert those UTC dates to local dates, which probably explains the relative slowness.
259-char path limit workarounds - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=26170
- (I had proposed A_LoopFileTimeModifiedUTC and A_LoopFileTimeCreatedUTC variables (and A_LoopFileTimeAccessedUTC for completeness), as they would be faster, but also more useful, when comparing files across DST and time zone differences.)
- (Similarly, an A_LoopFileAttribValue variable would be faster than A_LoopFileAttrib, as you'd get the raw number from the struct instead of converting the number to letters (from the list 'RASHNDOCT'), and more useful, as any unusual properties would also be listed.)

Re: jeeswg's benchmark tests

Post by jeeswg » 02 Jun 2018, 16:23

- I ran some tests against A_ variables:

Code: [Select all] [Expand]GeSHi © Codebox Plus


- Here are two batches of results for the slowest variables totalling 1 second or more for 1000 reads:

Code: [Select all] [Expand]GeSHi © Codebox Plus


- Summary of results:
very slow: A_IPAddress1/2/3/4
quite slow: A_IsAdmin, A_UserName
also slow: A_CaretX/Y, A_ComputerName, A_Cursor, A_Now/A_NowUTC
also slow: dir-related variables:
A_AppData/A_AppDataCommon
A_Desktop/A_DesktopCommon
A_MyDocuments
A_ProgramFiles
A_Programs/A_ProgramsCommon
A_StartMenu/A_StartMenuCommon
A_Startup/A_StartupCommon
A_Temp

- Note: All of the variables mentioned in WINDOWS FOLDER LOCATIONS, here:
jeeswg's Explorer tutorial - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=7&t=31755
appeared in the list of slow variables, apart from A_ComSpec and A_WinDir.
- Note: Clearly tests of file/registry loop variables should be done within loops for more useful results.

- Threads that prompted the tests:
A_LoopFileLongPath MUCH slower than A_LoopFileFullPath - Issues - AutoHotkey Community
https://autohotkey.com/board/topic/55613-a-loopfilelongpath-much-slower-than-a-loopfilefullpath/
Possible A_LoopFileLongPath bug - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=5&t=42891
A_IPAddress Performance - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=14&t=49910
- Note also:
Test build - Obj.Count(), OnError(), long paths, experimental switch-case - AutoHotkey Community
https://autohotkey.com/boards/viewtopic.php?f=24&t=47682
Fixes a significant performance issue with A_LoopFileLongPath (and A_LoopFileShortPath is also optimized).

Re: jeeswg's benchmark tests

Post by nnnik » 13 Mar 2018, 02:34

Have you compared the precision of the results?
That would be my only guess

Re: jeeswg's benchmark tests

Post by jeeswg » 13 Mar 2018, 02:07

Why do some people use the MulDiv function, instead of built-in operators?

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by nnnik » 25 Feb 2018, 15:29

@Helgef yes it is. However AutoHotkey itself will speed up after the first test. ( The low level code that handles all sort of stuff )
Thats consistent with our findings that the 1st test is the slower than the second test of the same type no matter what happens.
Depending on which things we do our branch prediction might already be trained for specific actions in autohotkey while it isn't trained for others.

Re: jeeswg's benchmark tests

Post by Helgef » 25 Feb 2018, 14:31

Use, quotes for the types to avoid creating unnecessary variables. #NoEnv is important when omitting quotes for types, as documented. Also, if dllcall performance is an issue, AHK_H has dynacall, which is generally faster if I am not mistaken.

@nnnik, I think script code is a little too high level for branch prediction have this effect.

Cheers.

Re: jeeswg's benchmark tests

Post by nnnik » 24 Feb 2018, 07:35

jeeswg wrote:- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.

Thats due to the way modern CPUs work - the thing you are looking for is called Branch Prediction.

Also after testing it myself it seems that using a pointer isn't faster than using the dll functions name.

Re: jeeswg's benchmark tests

Post by jeeswg » 24 Feb 2018, 07:00

- I'd be interested in any benchmark tests re. optimising DllCall results, e.g. specifying dll name or not, specifying .dll or not, specifying W/A or not, using Int or "Int" and anything else. I haven't been able to get any clear-cut results so far.
- The documentation does mention about using LoadLibrary and using a function address (for dlls that aren't pre-loaded).
DllCall
https://autohotkey.com/docs/commands/DllCall.htm#load
- Also, any good example dlls/functions for testing would be helpful, it's hard to find the best ones for testing.

- Also, doing a benchmark test, with the exact same code being tested twice in a row, with one being faster than the other, is concerning.

Re: jeeswg's benchmark tests

Post by jeeswg » 17 Feb 2018, 07:25

Here's an interesting one, object count keys: clone+delete v. for loop v. delete. See results at the bottom. Cheers.

Code: [Select all] [Expand]GeSHi © Codebox Plus

Re: jeeswg's benchmark tests

Post by jeeswg » 01 Feb 2018, 10:27

- I had thought that the for loop might be faster. Why did we get different results (did you perform the tests twice, swapping the order, did you perform the operation more times)? Why are earlier tests faster within the same script run, should we have a delay after a hotkey is executed or a dummy test?
- I've tried to incorporate every piece of advice I've been given. So getting incorrect results at this stage is really facepalm.
- Btw how do x64/x32 compare generally?
- Also I've been wondering about 'CPU' benchmark tests, or other measures, i.e. the method that takes less of a toll on the system.
- Yes, Helgef, it's milliseconds. I've now corrected it everywhere in this thread. [@nnnik: could you edit your post to make it say 'benchmark tests (ms)', thanks.]
QueryPerformanceCounter MSDN page:
Retrieves the current value of the performance counter, which is a high resolution (<1us) time stamp that can be used for time-interval measurements.
...
A pointer to a variable that receives the current performance-counter value, in counts.
QueryPerformanceFrequency MSDN page:
A pointer to a variable that receives the current performance-counter frequency, in counts per second.

Code: [Select all]GeSHi © Codebox Plus

;frequency = count / time
;count = frequency * time
;time = count / frequency
DllCall("kernel32\QueryPerformanceFrequency", Int64P,vFreq) ;count per sec
DllCall("kernel32\QueryPerformanceCounter", Int64P,vCount) ;count
MsgBox, % (vCount / vFreq) ;seconds
MsgBox, % (vCount / vFreq) * 1000 ;milliseconds

Re: jeeswg's benchmark tests

Post by Helgef » 01 Feb 2018, 05:35

for loop v. loop (on array)
loop slightly faster

For me for-loop is faster, as expected.

Cheers.


note: this function returns seconds, the original function returns milliseconds

Note: Your function returns milliseconds.

Re: jeeswg's benchmark tests

Post by nnnik » 01 Feb 2018, 05:06

Rather than leaving it to writing manual tests all the time we should probably write a benchmark library - that handles the tests correctly

Re: jeeswg's benchmark tests

Post by jeeswg » 01 Feb 2018, 04:53

Thanks nnnik, I had been concerned about whether order mattered. Does anyone have any ideas?

Top