Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

AutoHotkey Benchmarks


  • Please log in to reply
53 replies to this topic
Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010

This project aims to be a comprehensive reference for the speed of the most commonly used snippets of code. Here is the result:
 
 

All-in-one Benchmark (Uberi) (LATEST: 12/5/13)

All-in-one Benchmark (Thanks Guest!) 
(NOTE: Contains only AutoHotkey v1.0.48.05 and v1.1.08.01 ANSI)
 
All-in-one Benchmark (Thanks shajul!) (NOTE: Contains only AutoHotkey v1.0.48.05 and v1.0.92.02 ANSI + Unicode)
 
AutoHotkey v1.1.10.00 (Uberi) (LATEST: 12/5/13)

AutoHotkey v1.1.09.00 (Uberi)
AutoHotkey v1.1.08.00 (Uberi)
AutoHotkey v1.1.07.03 (Uberi)
AutoHotkey v1.1.06.01 (Uberi)

AutoHotkey v1.1.05.06 (Thanks Rseding91!)
AutoHotkey v1.1.05.06 (Thanks Rseding91!)
AutoHotkey v1.1.05.06 ANSI (Thanks Rseding91!)
AutoHotkey v1.1.05.06 ANSI (Thanks Rseding91!)
AutoHotkey v1.1.05.06 x64 (Thanks Rseding91!)
AutoHotkey v1.1.05.06 x64 (Thanks Rseding91!)

AutoHotkey v1.1.05.00 (Uberi)
AutoHotkey v1.1.04.01 ANSI (Uberi)
AutoHotkey v1.1.03.00 (Uberi)
AutoHotkey v1.1.02.00 ANSI (Uberi)
AutoHotkey v1.1.00.00 ANSI (Uberi)
AutoHotkey v1.0.97.02 ANSI (Uberi)
AutoHotkey v1.0.97.00 ANSI (Uberi)
AutoHotkey v1.0.96.00 ANSI (Uberi)
AutoHotkey v1.0.95.00 (Uberi)

AutoHotkey v1.0.92.02 (Thanks Nameless-js/Leo Xiong!)
AutoHotkey v1.0.92.02 (Thanks Rseding91!)
AutoHotkey v1.0.92.02 (x64) (Thanks Rseding91!)

AutoHotkey v1.0.48.05 ANSI (Uberi)
AutoHotkey v1.0.48.05 ANSI (Thanks Morpheus!)
AutoHotkey v1.0.48.05 ANSI (Thanks tidbit!)
AutoHotkey v1.0.48.05 ANSI (Thanks Rseding91!)
AutoHotkey v1.0.48.05 ANSI (Thanks Rseding91!)



Now available at GitHub

And here is the code used to generate these reports:

AutoHotkey Benchmarks

If a benchmark summary showing multiple versions is needed, this script can combine two benchmark summaries into one:

Combine Benchmarks

Submissions for other AutoHotkey versions (getting it to work on IronAHK would be nice biggrin.png) are welcome, and this post will be edited to contain the latest information.

Edit 1: Added new stylesheet, allowed selection of stylesheet from benchmarker.
Edit 2: Added benchmarks by tidbit.
Edit 3: Added benchmarks by rseding91, and referenced shajul's work.
Edit 4: New benchmark version! See below for details.
Edit 5: New stylesheet, "Desaturated", added. Only the links with "(Uberi)" or "(Thanks Nameless-js/Leo Xiong!)" are using it right now, although it has now been made the default.
Edit 6: All benchmarks updated to newest theme, "Desaturated", and added system information table where possible. As a result, all benchmarks have been moved to my AutoHotkey.net account.
Edit 7: Restyled shajul's work, and added system information category.
Edit 8: Added benchmark for newest version, AHK_L v1.0.95.00, released today. Fixed several bugs. See update lower in this thread for more details.
Edit 9: Added benchmark for newest version, AHK_L v1.0.96.00, released a few days ago.
Edit 10: Added a script to combine different benchmark summaries together, as well as the output.
Edit 11: Added benchmark for newest version, AHK_L v1.0.97.00, released today. Also fixed bugs in benchmark combiner, and updated combined benchmark.
Edit 12: Added benchmark for AHK_L v1.0.97.02, released a day or so ago.
Edit 13: Fixed mistake in combined benchmarks
Edit 14: Added benchmark for the newest version of AHK, AHK_L v1.1.00.00, released a few days ago.
Edit 15: Added benchmark for newest version of AHK and updated combined benchmarks. See relevant post for full details.
Edit 16: Added benchmarks for AHK v1.1.03.00.
Edit 17: Added benchmarks for AHK v1.1.04.01. Move to GitHub Pages complete.
Edit 18: Added benchmarks for AHK v1.1.05.00.
Edit 19: Added benchmarks for AHK v1.1.06.01.
Edit 20: Added Rseding91's benchmarks. Changed naming scheme to be more consistent.
Edit 21: Added Morpheus' benchmark.
Edit 22: Add benchmarks for AHK v1.1.07.03 and v1.1.08.00.
Edit 23: Add benchmarks for AHK v1.1.09.00 and updated combined benchmark.
Edit 24: Add benchmarks for AHK v1.1.10.00 and updated combined benchmark.

Edit 25: Fix all the broken links.



tidbit
  • Administrators
  • 2709 posts
  • Hates playing Janitor
  • Last active: Jan 15 2016 11:37 PM
  • Joined: 09 Mar 2008
My laptop test:
my results: <!-- m -->http://www.autohotke...it/results.html<!-- m -->
my laptop specs: <!-- m -->http://www.autohotke...Diag_Laptop.txt<!-- m -->

Nice test. and nice style sheet ;) cool glow.

rawr. be very afraid
*poke*
. Populate the AutoHotkey city. Pointless but somewhat fun. .


Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010

Nice test. and nice style sheet ;) cool glow.


More stylesheets coming soon! :D

Stay tuned...

Edit: Also, just wondering, how do you manage on < 500 mb of RAM? My Win7 system uses 560 mb when idle :shock:.

Rseding91
  • Members
  • 703 posts
  • Last active: Apr 02 2016 05:05 AM
  • Joined: 07 Jun 2010
These are the benchmarks from my desktop computer:

AHK Basic: Here
AHK L x86 Unicode: Here
AHK L x64 Unicode: Here

System specs:
Processor: Intel Core i7 CPU 860 @ 3.3GHz
Memory: 16,379MB RAM (4 x 4GB) DDR3 1333 MHz
Video Card: NVIDIA GeForce 8800 GTX Superclocked
Operating System: Windows 7 professional x64

shajul
  • Members
  • 571 posts
  • Last active: Aug 01 2015 03:45 PM
  • Joined: 15 Sep 2006
Benchmarking three flavors of AHK side by side for easy comparison..

View here -> Benchmarks | My system info

If i've seen further it is by standing on the shoulders of giants

my site | ~shajul | WYSIWYG BBCode Editor

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
Much improved version, with the following changes:

- Improved performance.
- Added general speed settings control.
- Added selection of styles by name.
- System information category! No need to include your system specs anymore.
- Iteration counter now shows total iterations.
Next to be implemented:

- A way of using more than one version of AutoHotkey at a time
Enjoy!

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
New stylesheet, "Desaturated", added and made default, as promised. Only the links with "(Uberi)" or "(Thanks Nameless-js/Leo Xiong!)" are using it right now. Still looking into way to benchmark more than one version at a time. Any suggestions?

Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006
Some observations:
[*:35s9bzif]The performance of many of the examples depends, to varying degrees, on the input. For example, how well expression assignments compare to traditional assignments may depend on the length of the string or the number of variable references. One method might be more efficient for script A but less efficient for script B, so the results provided by these synthetic benchmarks can't necessarily be relied upon.
[*:35s9bzif]The ternary example, % (Variable = "abcdef") ? "", is in fact a syntax error and probably not the best (or most common) use of that operator. Strictly speaking, since you are omitting the "else" branch, it is really a binary operation, not ternary at all.
[*:35s9bzif]The RegEx examples are too arbitrary and simplistic, probably not representative of the average use of those functions.
[*:35s9bzif]The FileAppend example, appending a single character, is definitely not representative of how that command is ordinarily used. Current builds of AutoHotkey_L are for more inefficient with longer text. (My current development build improves on this considerably.)
[*:35s9bzif]Benchmarking Loop `n Return will tell you how efficiently the loop is set up, but how often do you really stop a so-called "loop" at its first iteration? The setup overhead of a Loop becomes less and less significant the more iterations there are.

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010

Some observations:
[*:1adhb2fv]The performance of many of the examples depends, to varying degrees, on the input. For example, how well expression assignments compare to traditional assignments may depend on the length of the string or the number of variable references. One method might be more efficient for script A but less efficient for script B, so the results provided by these synthetic benchmarks can't necessarily be relied upon.
[*:1adhb2fv]The ternary example, % (Variable = "abcdef") ? "", is in fact a syntax error and probably not the best (or most common) use of that operator. Strictly speaking, since you are omitting the "else" branch, it is really a binary operation, not ternary at all.
[*:1adhb2fv]The RegEx examples are too arbitrary and simplistic, probably not representative of the average use of those functions.
[*:1adhb2fv]The FileAppend example, appending a single character, is definitely not representative of how that command is ordinarily used. Current builds of AutoHotkey_L are for more inefficient with longer text. (My current development build improves on this considerably.)
[*:1adhb2fv]Benchmarking Loop `n Return will tell you how efficiently the loop is set up, but how often do you really stop a so-called "loop" at its first iteration? The setup overhead of a Loop becomes less and less significant the more iterations there are.


Good points, Lexikos, I will try to address them individually:

- Although synthetic benchmarks may not be truly representative of real-world code, it is a good way of comparing how snippets of code similar in function compare against each other in terms of performance, in the average script. In other words, it's not possible to use real-world examples, and expect comparisons to be fair. Also, a benchmark for variable references in "Classic" and "Expression" assignment has been added.
- That's a syntax error? I never knew :oops:. What would be a side-effect free way of writing that then? Also, the "Else" branch has been added.
- I do agree that the regular expressions were a bit too simple. They were originally part of the "String Searching" category, and so were meant to be functionally identical to InStr() and the like. The regular expression benchmarks now use regular expressions that are more representative of real-world usages.
- That particular benchmark was meant to show the overhead of using FileAppend once. Unfortunately, any longer texts would begin to degrade the validity of the benchmark, as hard drive speeds can vary considerably. In fact, it is rather inaccurate already, although that can't be helped.
- The loop benchmark was added solely for the purpose of comparison with other types of loops. Similar to the control run, it allows us to see the real overhead in the other types of loops, such as the overhead of parsing a string in a parsing loop.

Script and some benchmarks have been updated. Additionally, all completed benchmarks will be transitioned to the new theme.

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
More updates!

- New theme now used for all benchmarks
- Added system information table, where possible
- Moved all files to my AutoHotkey.net account, for easier modification
Edit: I forgot to add shajul's work :oops:. It will be up as soon as possible.
Edit: Added!

Lexikos
  • Administrators
  • 9844 posts
  • AutoHotkey Foundation
  • Last active:
  • Joined: 17 Oct 2006

Although synthetic benchmarks may not be truly representative of real-world code, it is a good way of comparing how snippets of code similar in function compare against each other in terms of performance, in the average script.

I think you missed my point. Some of those synthetic benchmarks are not representative of "the average script."

In other words, it's not possible to use real-world examples, and expect comparisons to be fair.

Why? If the real-world examples are closer to average use, they should be fairer.

What would be a side-effect free way of writing that then?

Simply adding the "else" branch allows the operation to complete rather than prematurely aborting.

Unfortunately, any longer texts would begin to degrade the validity of the benchmark, as hard drive speeds can vary considerably.

Benchmark results are not "portable" to begin with; i.e. the result of benchmark A on system X cannot be compared to the result of benchmark B on system Y. If benchmark A and benchmark B both write the same amount of data to the same hard drive, hard drive speed is irrelevant.

For instance, v1.0.92.02 vs v1.0.48.05 indicates that FileAppend c takes 80% longer in AutoHotkey Basic, but if I run both tests myself, I see that v1.0.92.02 actually takes 50% longer. My own PC completes the test with v1.0.48.05 about six times faster than your PC.

Furthermore, as I pointed out, current builds of AutoHotkey_L are inefficient with longer text; this has nothing to do with hard drive speed. For instance, appending "c", there appears to be very little difference between the current build and my latest test build, whereas appending "Lorem ipsum dolor sit amet" shows a difference of about 25% between the two builds. If I copy a 15MB ANSI-encoded log file line by line, my test build completes about 50 times faster than the current build, and most of that difference is in FileAppend.

The loop benchmark was added solely for the purpose of comparison with other types of loops. Similar to the control run, it allows us to see the real overhead in the other types of loops, such as the overhead of parsing a string in a parsing loop.

By terminating the loop early on its first iteration, you are skipping some of the overhead.

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010

Unfortunately, any longer texts would begin to degrade the validity of the benchmark, as hard drive speeds can vary considerably.

Benchmark results are not "portable" to begin with; i.e. the result of benchmark A on system X cannot be compared to the result of benchmark B on system Y. If benchmark A and benchmark B both write the same amount of data to the same hard drive, hard drive speed is irrelevant.


Actually, I was referring to certain events such as an OS cache miss, or a sudden disk seek caused by another application. This would introduce up to 10 milliseconds of inaccuracy, depending on the drive.

For instance, v1.0.92.02 vs v1.0.48.05 indicates that FileAppend c takes 80% longer in AutoHotkey Basic, but if I run both tests myself, I see that v1.0.92.02 actually takes 50% longer. My own PC completes the test with v1.0.48.05 about six times faster than your PC.


Yes, I realize that they cannot be compared between computers (other than shajul's benchmark, in which all 3 were run on the same computer), as they are simply meant as a reference between different coding styles and methodologies. A solution for comparing benchmarks between computers is in progress, using multiple versions of AutoHotkey included with the script.

Furthermore, as I pointed out, current builds of AutoHotkey_L are inefficient with longer text; this has nothing to do with hard drive speed. For instance, appending "c", there appears to be very little difference between the current build and my latest test build, whereas appending "Lorem ipsum dolor sit amet" shows a difference of about 25% between the two builds. If I copy a 15MB ANSI-encoded log file line by line, my test build completes about 50 times faster than the current build, and most of that difference is in FileAppend.


So you are suggesting the amount of text being appended be increased, in order to more accurately represent actual usages of this command, correct?

By terminating the loop early on its first iteration, you are skipping some of the overhead.


Loop tests modified.

nimda
  • Members
  • 4368 posts
  • Last active: Aug 09 2015 02:36 AM
  • Joined: 26 Dec 2010
It would be interesting to see the differences between:
var2 := var1
and
var2 = %var1%


Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
I'm guessing it would be something similar to this benchmark. I will add it in future benchmark runs though, thanks for the idea!

Uberi
  • Moderators
  • 1119 posts
  • Last active: May 02 2015 06:05 PM
  • Joined: 23 Aug 2010
Update:

[*:2u9x8gzv]AHK_L v1.0.95.00 was released today! Benchmarks were run again. New submissions are appreciated.
[*:2u9x8gzv]Fixed bugs in system information aggregator, now works on all major AHK versions.
[*:2u9x8gzv]Minor display changes.