Exo: JavaScript unleashed!

[Aurelain]

Exo
License: WTFPL
Imagine the power of AHK (File/OS/Keyboard/Mouse/GUI Management) in the hands of JavaScript.
Imagine the ecosystem of JavaScript libraries (jQuery, CreateJS, etc.) in the hands of AHK.
Exo makes this happen!

Description
Exo exposes most built-in commands/functions/variables of AHK to JavaScript, allowing you to practically write AHK inside JS (with JS syntax).

There are several advantages to using JS, rather than AhkScript:

1. A lot of people are already familiar with JS, so there's no need to learn the peculiarities of the AhkScript-syntax.
2. Gain access to a huge repository of libraries written in JS.
3. You can use JS-specific constructs (eval, anonymous functions, prototype, etc.).
4. You get to code in your favorite JS editor (IDE).

... and more.

It does all this by embedding a browser (Internet Explorer) and hooking into it with ComObject. It then bridges the JS global functions to their AHK counterpart.

Goals
While migrating the keywords, the following goals were set:

• try to port 100% of the functionality (89% achieved)
• try as much as possible to conform with the AHK documentation
• provide useful typing (i.e. we shouldn't return String types everywhere only because it's easy)

Usage
Launch Exo with a ".js" file argument (either through drag-and-drop, or from the command-line):

> Exo.ahk "demo.js"

Sample code:

Code: Select all

alert(A_Now);
FileAppend('Hello, World!', 'hello.txt')
var list = Loop('*.txt');
for (var i = 0; i < list.length; i++) {
	var fileProperties = list[i];
	MsgBox(fileProperties.LongPath);
}
Sleep(1000);
ExitApp();

Migration Guide
The GitHub readme lists all relevant AHK keywords (Commands, Function, Directives and Variables) and illustrates their counterpart in JS. In most cases, there is no significant change and everything is intuitive.

The breakdown for the 470 keywords:

• 0) 13% were disregarded because they provided duplicate functionality
• 1) 37% were directly migrated (AHK Functions became JS Functions, AHK Variables became JS Variables)
• 2) 33% were migrated through subtle conversion (AHK Commands became JS Functions)
• 3) 6% were migrated through minor adjustment (AHK multiple-OutputVars became JS return Objects)
• 4) 7% could not be migrated, but there are alternatives or future prospects of being implemented.
• 5) 4% will never be migrated (mainly Directives and some very specific AHK Commands)

Source: Exo on GitHub (you also need the items in the "lib" sub-folder).

[Aurelain]

Notes:

• Shared credits:
  • lexikos, for putting me on the right track in this thread
  • Coco, for the ComDispatch library
  • GeekDude, for the FixIE function
  • SKAN, for the Args function
• The RegEx is faster in JS than in AHK. For certain kinds of scenarios, it might be better to write your script in JS, and output the results with Exo.
  However, the overhead of moving large chunks of data in and out of JS usually is larger than the gains of the faster RegEx.
  Besides, I find it hard to think of a circumstances where these speed differences matter.
• Errors thrown in AHK produce warnings in JS and can be caught with try/catch in JS (I couldn't believe it when I saw it). However, Exo denies this mechanism by enclosing the AHK code in a try/catch of its own. It does this to stop JS producing popups for errors that are normally silent (e.g. a FileDelete that fails).

[joedf]

Very Nice! Gives like a feeling for Node.js or PhantomJS... Nicknames : ahk.js or js.ahk

[lexikos]

Looks good. I'll be interested to see how much this is taken up.

It does all this by embedding a browser (Internet Explorer) and hooking into it with ComObject.

That seems a bit... heavy. Why not use the ScriptControl object or ActiveScript? You wouldn't get functions like eval(), I suppose.

[joedf]

Yes, I actually thought of that too, but wasn't sure...
By the way, happy holidays! (@lexikos it's alright, no need to respond )

[lexikos]

You've marked EnvAdd and EnvSub as "Not needed (Feature duplication)", but does JS have date math? If not, I suppose you could implement v2's DateAdd and DateDiff using EnvAdd and EnvSub respectively, rather than duplicating those commands exactly.

The RegEx is faster in JS than in AHK.

They have slightly different syntax and features, though. Using AutoHotkey's RegEx (PCRE) within JS would probably be inconvenient, due to the need to escape backslashes, which doesn't apply to /this syntax/.

[Aurelain]

Gives like a feeling for Node.js

Indeed, it was Node.js that put me on this path.
However, I disliked its emphasis on asynchronous operations (they do offer synchronous file operations, but they're like an afterthought).
Anyway, AHK (and by extension Exo) is richer, leaner and no-nonsense .

Why not use the ScriptControl object or ActiveScript?

Because I bulled through with development and did not investigate alternatives .
I'll research ActiveScript now, but if I were to defend the embedded browser, here's a quick argument:

• You can also build actual HTML UI and "sprinkle" it with Exo.

I'm also curious to see how much Exo is taken up. I can say one thing for sure: I'll convince people at my workplace to use it.

[vasili111]

Very good idea. JavaScript is very powerful language with lots of libraries.
By the way, DRAKON Editor suppurts JavaScript too.

[Aurelain]

About EnvAdd and EnvSub...
JS does allow operations with dates.
They are a bit cumbersome and don't support the YYYYMMDDHH24MISS format, so there is a loss of functionality here.
I'll add DateAdd and DateDiff on the TODO list.

About the RegEx...
I've attached a quick comparison.
I used a large XML file (80MB, needed to add #MaxMem 1000), and used the following test units:

Spoiler

Code: Select all

FileRead, contents, large_file.xml
time := A_TickCount
output := RegExReplace(contents, "<.*?>", "")
Msgbox % A_TickCount - time ; 1200

Code: Select all

var contents = FileRead("large_file.xml");
var time = new Date().getTime();
contents = contents.replace(/<.*?>/g,'');
alert(new Date().getTime() - time); // 1400

Code: Select all

var contents = FileRead("large_file.xml");
var time = new Date().getTime();
contents = RegExReplace(contents, "<.*?>", "");
alert(new Date().getTime() - time); // 1800

Conclusions:

• I was wrong about JS RegEx being faster than AHK RegEx (or at least it's not clear cut)
• Escaping is not annoying in Exo RegEx. As a side-note, escaping DOS-style paths in Exo is annoying (FileRead("C:\\Windows\\system.ini")).
  That is why I added the following TODO note: "Accept/Provide URI paths in all transactions (not DOS-style paths)"

[lexikos]

It looks like you've worked around a number of limitations which actually don't exist.

Exo.send stores closures and instead passes an ID to AutoHotkey. This is unnecessary: you can simply pass the closure, and later call closure.call(...) from AutoHotkey. That also allows you to specify this (or if you pass ComObject(0,0), it automatically gets replaced with the window object).

trigger (in Exo.ahk) has an if-else-if ladder to handle varying number of parameters. Instead, you can do this:

Code: Select all

result := ExoObj.trigger(closureId, args*)

The description of Exo.receive reads:

* This function gets called by AHK to store values in "_result".
* It was needed because we want the result to have a native type, not just "String".
* The "_result" that gets set here will be returned by "send()".

This is not true: ComDispatch is capable of returning not just strings, but also numbers and COM objects. However, your inject and receive functions still have a purpose: they convert AutoHotkey arrays to JavaScript arrays, and likewise for Objects. One alternative would be for the API functions to create JavaScript arrays/objects (via some helper JS) and return those directly.


I'm not really sure what you mean by "attempt to mitigate the race condition by speeding up the registering", but I always wait for the document to load (even about:blank):

Code: Select all

while wb.ReadyState != 4
    Sleep 100

fixIE (or your usage of it) has problems:

• It writes to the registry, then later deletes the value, with no regard to whether the value already existed (i.e. perhaps the user already set a value for AutoHotkey.exe to cover all of their scripts).
• Your script never gets to call fixIE(false) if the js called ExitApp. You could work around that by calling fixIE(false) before writing the mainContent script. (By this time, the rendering mode is already locked in, so the registry setting is no longer needed.)

I'm not sure if it's completely equivalent, but at least going by document.documentMode (and the fact that Exo works), using the X-UA-Compatible tag is an alternative. To use it, you must load the document from a file (see below).

I think it's bad form to document.write things before and after the <html> tags, or at all. You can eliminate fixIE and one document.write call by using a HTML file:

Code: Select all

<!doctype html>
<html>
<head>
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <script src="lib/Exo.js"></script>
</head>
<body>
</body>
</html>

If Exo was compiled, this HTML could be embedded in the EXE as a resource and loaded via the res:// protocol. The same could be done for Exo.js, so no external files would be needed. (Personally, I'd prefer not to compile Exo.)

Instead of writing <script>__AHK = {}</script>, you can just include var __AHK in Exo.js and set __AHK after the document finishes loading.

Instead of writing "<script>" . mainContent . "</script>", you can call wb.document.parentWindow.eval(mainContent).


With AutoHotkey v1.1.17 (released today), ComDispatch is not needed. You can replace the first line below with the second:

Code: Select all

wb.document.parentWindow.__AHK := ComDispatch("", "bridge")
wb.document.parentWindow.__AHK := {bridge: Func("bridge")}

It's also possible to pass Func objects to JS directly. So for instance:

Code: Select all

; Put this in Exo.js:
var bridge

; After the document loads, initialize bridge like so:
wb.document.parentWindow.bridge := Func("bridge")

; Also remove 'this' from the parameter list of bridge:
bridge(name,args*){

[Aurelain]

Lexikos, thank you so much for the review.
To sum-up your post:

1. Exo could pass native closures (JS functions) directly to AHK so they can be triggered at a later time by AHK. I've tested it and you're right. It's pretty close to magic how something like this can be achieved . I will soon implement it, but first let's debate an issue.
   
   The current setup relies on a single function ("send()") to communicate with AHK, by passing arguments through "eval()".
   Basically, it stringifies the arguments, which is something that prevents us sending actual closures.
   We should therefore "apply()" the "bridge()". But we can't, because in JS the "bridge()" has a strange existance:
   
   With "ComDispatch": typeof bridge -> "unknown", JSON.stringify(bridge) -> "undefined"
   With "Func(bridge)" (new in v1.1.17.00): typeof bridge -> "Object", JSON.stringify(bridge) -> "{}"
   
   So I think we're left with 2 options:
   A. either I rethink the whole JS API and no longer use a centralized "send()", but instead each JS API function directly calls "bridge()" with whatever arguments it needs
   B. or I create an if-else-if ladder (better yet, a switch) and make 10 direct calls to "bridge()" inside "send()", based on the number of arguments.
   I'd strongly go with option B. What's your take on this?
2. AHK is capable to return not just strings, but also numbers.
   However, as you've mentioned, inject/receive still have a role in returning Array/Object.
3. We should wait for the browser's ReadyState.
   You're right about it. Will do.
4. FixIE is used somewhat crudely.
   Will solve it.
5. The current usage of document.write could be replaced through more conventional methods.
   Will investigate, I think I tried X-UA-Compatible a month ago and failed. Either way, writing outside the <html> tag is something I also hate.
6. AutoHotkey v1.1.17 adds some syntactic sugar .
   I'd implement it in a heartbeat, but I'm worried not enough people have it. Should I go for it?

[Coco]

Here's a mod of FixIE() (works on v2 as well):

Code: Select all

/* Function: BrowserEmulation
 *     FEATURE_BROWSER_EMULATION -> http://goo.gl/V01Frx
 * Syntax:
 *     BrowserEmulation( [ version ] )
 * Parameter(s):
 *     version  [in, opt] - IE version(7+). Integer between 7 - 11, value can be
 *                          negative. If omitted, current setting is returned.
 *                          If 'version' is the word "edge", installed IE version
 *                          is used. See table below for possible values:
 * Parameter "version" values table:
 * (values that are negative ignore !DOCTYPE directives, see http://goo.gl/V01Frx)
 *     7:  7000               ; 0x1B58           - IE7 Standards mode.
 *     8:  8000,  -8:  8888   ; 0x1F40  | 0x22B8 - IE8 mode, IE8 Standards mode
 *     9:  9000,  -9:  9999   ; 0x2328  | 0x270F - IE9 mode, IE9 Standards mode
 *     10: 10000, -10: 10001  ; 0x02710 | 0x2711 - IE10 Standards mode
 *     11: 11000, -11: 11001  ; 0x2AF8  | 0x2AF9 - IE11 edge mode
 * Remarks:
 *     This function modifes the registry.
 */
BrowserEmulation(version:="*") ;// FEATURE_BROWSER_EMULATION -> http://goo.gl/V01Frx
{
	static key := "HKCU\Software\" . ( A_Is64bitOS ? "Wow6432Node\" : "" )
	. "Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BROWSER_EMULATION\"
	. ( A_IsCompiled ? A_ScriptName : StrGet(DllCall("Shlwapi\PathFindFileName", "Str", A_AhkPath)) )

	static WshShell := ComObjCreate("WScript.Shell")
	
	static prev := "*"
	if (prev == "*") ;// init
		try prev := WshShell.RegRead(key)
		catch ;// value does not exist
			prev := ""

	;// __Get()
	if (version == "*")
	{
		try val := WshShell.RegRead(key)
		catch ;// value does not exist
			val := "" ;// throw Exception()??
		finally
			return val
	}

	;// __Set()
	ie_version := Round(WshShell.RegRead("HKLM\SOFTWARE\Microsoft\Internet Explorer\svcVersion"))
	if (ie_version < 8) ;// unsure of this but I read it somewhere...
		throw Exception("FEATURE_BROWSER_EMULATION is only available for IE8 and later.")

	if version
	{
		if !(version ~= "i)^7|-?(8|9|1[01])|edge$")
			throw Exception(Format("Invalid argument: '{1}'", version))
		if (version = "edge")
			version := ie_version
		version := Round(version * (version>0 ? 1000 : -(version>=-9 ? 1111 : 1000.1)))
	}

	try version? WshShell.RegWrite(key, version, "REG_DWORD")
	           : (prev != "" ? WshShell.RegWrite(key, prev, "REG_DWORD") : WshShell.RegDelete(key))
	catch error
		throw error
	finally
		return BrowserEmulation() ;// OR return the value itself
}

Specify the version(7-11) for the argument, if version is negative (such as -11), the function will use the value w/c ignore !DOCTYPE directives. If version is the word edge, value is set based on the version of IE installed. Specify 0 or "" to disable(if the registry value already exists as pointed out by lexikos) OR delete the registry value.

[lexikos]

The current setup relies on a single function ("send()") to communicate with AHK, by passing arguments through "eval()".
Basically, it stringifies the arguments, which is something that prevents us sending actual closures.

No, that's not the case. It does not stringify the arguments; it writes a string of expressions: a[0], a[1], etc. When these are evaluated, they return the actual value of each array element, which is then passed on to the function. They can most certainly be objects.

We should therefore "apply()" the "bridge()". But we can't, because in JS the "bridge()" has a strange existance

Simply put, bridge is an AutoHotkey Func object, not a JavaScript Function object. With ComDispatch it was more strange: __AHK.bridge wasn't an object at all, but a member of an object.

This seems to work, even though bridge is not a JavaScript Function:

Code: Select all

Function.prototype.apply.call(bridge, null, ["MsgBox", "Hello, world!"])

Specify 0 or "" to disable(if the registry value already exists as pointed out by lexikos) OR delete the registry value.

It will not restore the previous setting, will it?

[lexikos]

There's a bug in OnMessage:

mapOnMessageClosures.Remove(MsgNumber)

When used like this, Remove interprets MsgNumber like an array index. If you have {100:a, 101:b} and you call Remove(100), you'll end up with {100:b}. You need to write Remove(MsgNumber, "") in v1.1 (not v2).

[Coco]

It will not restore the previous setting, will it?

Oops, fixed now. Thanks.

[Aurelain]

The following post is mostly of interest to lexikos.
I've tested the performance of large data transfers between AHK and JS.
The results are somewhat non-statistical, but there might be some usefull data here.

Basically, there are 3 ways of hooking into the ActiveX Internet Explorer:

1) Using ComDispatch (the way Exo currently uses)
wb.document.parentWindow.dummy := ComDispatch("", "bridge")

2) Registering an AHK object, with the newly added feature of v1.1.17.00:
wb.document.parentWindow.dummy := {bridge:Func("bridge")}

3) Registering an AHK function, with the newly added feature of v1.1.17.00:
wb.document.parentWindow.dummy.bridge := Func("bridge")

Also, there are 3 ways of obtaining data from AHK

A) The direct way: by using the return value.
var result = dummy.bridge();

B) The indirect way: AHK doesn't return anything, but instead sets a global variable in JS.
var result;
dummy.bridge(); // AHK sets "result" like this: wb.document.parentWindow.result = "foobar";

C) The convoluted way: AHK doesn't return anything, but instead calls a global function in JS that sets a global variable in JS.
var result;
var store = function(param){result = param;}
dummy.bridge(); // AHK calls "store()" like this: wb.document.parentWindow.store("foobar");

The test script is attached (not really interesting for anyone).
Large files can be found here
The results are attached. They vary from one run to another, but some things are constant:

• Using the return value is twice as slow as the other options
• The return value of ComDispatch has an atrocious performance
• Apparently, using Func() (combination C3) is best for Exo's purposes, but I'm not absolutely sure about this.

[lexikos]

Why not use the ScriptControl object or ActiveScript? You wouldn't get functions like eval(), I suppose.

That was a bad example; eval() is actually part of the JScript engine, not the IE object model. alert() is a better example. However, I've found that ScriptControl and ActiveScript aren't just lacking the alert function, they're also stuck with an older version of the JScript language.

Passing {16d51579-a30b-4c8b-a276-0ff4dc41e755} to ActiveScript or cscript.exe gets version 11 of the engine (according to ScriptEngineMajorVersion()), but it doesn't act like it. Date.now() (IE9) is available, yet JSON (IE8) is not. let and const (IE11) don't work.

These all work with your script, so using the WebBrowser control appears to be a good choice. An alternative would be to use the JavaScript Runtime Host, which:

• Has less overhead (no WebBrowser control or MSHTML).
• Doesn't need workarounds like fixIE or X-UA-Compatible.
• Requires IE11.

[Aurelain]

I analyzed lexikos's suggestion of flowing data directly between JS and AHK.

Metod A (the current way)

1. JS: an Exo function is called (e.g. FileRead())
2. JS: FileRead() calls a centralized send() function
3. JS: send() calls a centralized bridge() function, which is in fact a registered AHK function
4. AHK: bridge() calls the FileRead() AHK function
5. AHK: FileRead() calls the FileRead command. It then calls the inject() AHK function.
6. AHK: inject() calls the receive() JS function
7. JS: receive() writes the value into _result
8. JS: send() returns the value of _result
9. JS: FileRead() returns the value of send()

Advantages:

• It satisfies all typing requirements

Disadvantages:

• It requires a large ammount of boilerplate code
• It's difficult to understand
• It impacts performance by ~10%

Metod B (proposed by lexikos)

1. JS: an Exo function is called (e.g. FileRead()) which is in fact a registered AHK function
2. AHK: FileRead() calls the FileRead command and returns the Output if it is String or Number.

However, there are 15 functions that need to return Object or Array (ImageSearch, Loop, RegExMatch, etc.). In such cases, we have to go through some extra steps:
(3.) AHK: Call an utility function in JS (let's say getNativeObject())
(4.) JS: getNativeObject() returns an native Object
(5.) AHK: FileRead() returns the native Object.

Advantages:

• It's easy to understand

Disadvantages:

• We must actually register all supported AHK keywords (350). Currently, we're only handling 180 keywords, the rest being dynamically handled by bridge(). That means:
  • We have to create 170 extra functions in AHK
  • We have to register 350 AHK functions (currently only 1 is registered, the bridge())
• Using the return value is a bit slower than injecting values (see my previous post about this)
• Not having the centralized bridge(), we can't easily set the thread defaults.
• We cannot return undefined (!). This is close to a dealbreaker. The only thing we could do is to accept the situation and know that all functions that should return undefined will in fact return "".

It's obvious Method A is pretty strange, but I think I'll stick with .

[lexikos]

ComObject(0,0) corresponds to undefined, and ComObject(1,0) corresponds to null.

Regarding registering lots of functions, if you register the AutoHotkey functions directly with JavaScript, you won't need the 240 boiler-plate functions in Exo.js. I'm not sure why you think 170 extra functions would need to be created; are you talking about the ones that are handled dynamically by bridge()? Built-in functions can just be registered with JS directly, like so:

Code: Select all

w := wb.document.parentWindow
functions = Chr,Asc
Loop Parse, functions, `,
	w.eval("var x" A_LoopField), w["x" A_LoopField] := Func(A_LoopField)
; (I've added x so as not to conflict with those defined in Exo.js)

The eval/var declaration is due to AutoHotkey's lack of support for IDispatchEx. Calling IDispatchEx via DllCall is one alternative. Utilizing a JavaScript helper function is another.

Creating a native JavaScript Object doesn't require creating a helper function in JavaScript; one already exists. However, window.Object.create(ComObject(1,0)) is a bit verbose, and once you get the object, you still can't assign values to it without using IDispatchEx or a helper function.

Why do you set the thread defaults every time bridge() is called? Are they even thread-specific anymore?

In AutoHotkey v2, since all commands also exist as functions, you would only need to wrap them when you want to change the parameters or return value.

[Aurelain]

The arguments start to pile-up in favor of your suggested Method B. For example:

Why do you set the thread defaults every time bridge() is called? Are they even thread-specific anymore?

I just tested it and with the old ComDispatch, each function was indeed a new thread.
Now, with the new direct "Func()", everything works properly!

Code: Select all

Gui Add, ActiveX, vwb, Shell.Explorer
wb.Navigate("about:blank")
wb.document.parentWindow.eval("var Exo = {change:'', verify:''};")

old := false
if (old) {
	#Include %A_ScriptDir%\lib\ComDispatch.ahk
	#Include %A_ScriptDir%\lib\ComDispTable.ahk
	#Include %A_ScriptDir%\lib\ComVar.ahk
	wb.document.parentWindow.Exo := ComDispatch("", "change,verify")
} else {
	wb.document.parentWindow.Exo.change := Func("change")
	wb.document.parentWindow.Exo.verify := Func("verify")
}

wb.document.parentWindow.eval("Exo.change(1);Exo.verify(1);")
; The output with old: 1 (the thread default, meaning the previous "change()" was lost)
; The output with new: 2 ("change()" worked!)
ExitApp

change(t){
	SetTitleMatchMode, 2
}
verify(t){
	MsgBox % A_TitleMatchMode
}