65 replies to this topic

[PhiLho]

[UPDATE] SciTE4AHK.zip (around 775KB) contains the latest (or recent... v.3.2.1 at the time of writing, can be silently updated) version of SciTE.exe and SciLexer.dll with AHK support.
It also has various files to rebuild it yourself if needed (with Visual Studio) and properties, scripts (Lua, AHK...), etc. I use with SciTE.
See Update 1 and Update 2 for details.

Practical area
This is a lexer that allow proper syntax highlighting of AutoHotkey script in Scintilla based editors, like SciTE, Notepad2, Notepad++ (creative names...), etc.



If your editor has a file named SciLexer.dll, you can download my version (see above, now) to test it. Otherwise, either it doesn't use Scintilla or it uses a custom version (Notepad++), or it integrated it within the exe (Notepad2). You can still test my lexer, if you are curious, by downloading SciTE (the full download, not Sc1). Warning: setting up SciTE isn't very easy (you have to read the manual and edit a text file - nothing really exotic for an AHK user! :-)), although somebody made a GUI to handle the most common settings.
Backup (rename) the original SciLexer.dll, although this copy is reasonably up-to-date (at time of writing!), and replace it with my copy.

You also need the ahk1.properties
file.
Modify it to your needs/tastes (see below for the font & color definitions it uses). Don't forget to change SciTEUser.properties (or SciTEGlobal.properties if you are as foolish as me :-)):
open.filter=\
[...]
$(filter.ahk)\
[...]

menu.language=\
[...]
AutoHotkey|ahk||\
[...]

and, of course:
import ahk1


Release history

2006-11-26

* I renamed the lexer to ahk1 too, so now I renamed the properties file to ahk1.properties. This way, I am consistent all the way!

* I also corrected some minor bugs in the regex engine.

2006-11-14

* No changes on the lexer, but I updated the Scintilla version to the latest (1.71). I also improved the regular expression engine to support \d \D \s \S \w \W \xHH (HH = 2 hex digits) inside and outside char classes [...]. Beware for this one: I tested it, but I might have missed a bug. Not yet official.

* I also renamed the source from LexAHK.cxx to LexAHK1.cxx (I will rename also the lexer later), because I feel a LexAHK2.cxx will be necessary when the v.2 of AutoHotkey will be out!

2006-05-10

* Corrected the following bugs, and some others
- ::btw::By The Way <- No as string
- :*:²k:: ; Comment is still in string style

* Folding works!
I plan to add folding between special comments: ;{ and ;}
I might, if not too hard, add folding between a label (or hotkey/hotstring) and some standalone ending command (Return, Exit[App], others?).


Initial release notes
Finally, with quite some hard work, I have made a lexer for Scintilla that is worth making public.
This is the hardest lexer I had to make up to now, and it is not finished yet.
I have tested it some days, trying with my scripts and some 3rd party ones, and it is giving quite good results already.
So I release it, to be tested by other people, hoping to get feedback and bug reports.
After a while, I will submit it officially to be integrated to the Scintilla code.

This lexer has some bugs and limitations. I plan to address some of them in the future, but others will never be addressed, because of complexity, or because it would need solutions outside the scope and philosophy of the current lexers.

Before expanding on the problems, I will first present the working features! :-)

My lexer recognizes a number of syntax constructs:
(Legend: the number is the style number, the ^ symbol is used to mark a "beginning of line" in AutoHotkey's sense, ie. after skipping initial white space.)

0 - Base (background, base font)

32 - Default (everything not below: spaces, untyped parameters)

1 - Line comment
; Foo (must be preceded by a space after some code)

2 - Block comment
^/* ^*/

3 - Escape sequence
`n `, `% etc.

4 - Syntax operator
comma, variable assignment, continuation symbols, block symbols, "hot" colons, variable reference
, = := += -= /= *= ^( ^) { } : :: %

5 - Expression operator
( ) + - * ** / // ! ~ ^ & << >> . < > <= >= = == != <> && ||

6 - String
"" ^( ^) (continuation section)

7 - Number
0 -1 3232 5.914159 0xBEEF

8 - Identifier (variable & function call)

9 - Variable reference (dereferencing)
%varName%

10 - Label, Hotkeys & Hotstrings (& Function definition?)
foo: F4:: x::a ::btw::BTW :*:$$::dollar

Keywords
11 - Flow of control
12 - Commands
13 - (Built-in) Functions
14 - Directives
15 - Keys & buttons
16 - (Built-in) Variables
17 - "Keywords" (special parameters)
18 - User defined

19 - Variable keyword (built-in) dereferencing %A_xxx%

20 - Unterminated (expression) string, missing ending percent sign

In addition to the above ahk1.properties file, you may want to use my own color and font definitions (might be outdated...), to get an idea on how my choices look like (not all are relevant to AHK... Yet, I removed the GTK+ settings).

# Used when switching to monospace display (Ctrl+F11)
font.monospace=font:Lucida Sans Typewriter,size:8

# Monospace font for code
# Used for most code (no o0OIl1i confusion).
font.monospace.code=font:Andale Mono,size:8

# Monospace font for comments (boxed, with tables, alignments, etc.)
font.monospace.comment=font:Courier New,size:8

# Used to display rather informal/small code (scripts)
font.proportional.serif=font:Georgia,size:8
font.proportional.sans=font:Verdana,size:8

# A bit of fantasy...
font.fantasy=font:Comic Sans MS,size:8

# Readable, used for texts (docs, HTML text...)
font.text=font:Book Antiqua,size:9
font.text=font:Bitstream Vera Serif,size:9
font.text=font:Georgia,size:9


# For SciTE console
font.small=font:Arial,size:7

# Special uses (line numbers, UUID, etc.)
font.computer=font:Arial Narrow,size:9
font.computer=font:OCR A Extended,size:9

# I put this one in styles not used by the lexer (to be sure they are not used...)
font.notused=font:Webdings,size:7


#-- Font use independent of platform

# Script/small, informal code
font.base=$(font.proportional.sans)
# Regular code : monospace
font.code.base=$(font.monospace.code)
# Comments, proportional in line comments (rarely aligned),
# fixed in stream comments (often boxed)
font.code.comment.box=$(font.monospace.comment)
font.code.comment.line=$(font.proportional.serif)
font.code.comment.doc=$(font.monospace.comment)

font.text.base=$(font.text)
font.text.comment=$(font.fantasy)

font.embedded.base=$(font.base)
font.embedded.comment=$(font.proportional.serif)

## Give symbolic names to the set of colours used in the standard styles.
colour.code.comment.box=fore:#7070A0
colour.code.comment.line=fore:#8080A0
colour.code.comment.doc=fore:#0000A0
colour.text.comment=fore:#0000FF,back:#D0F0D0
colour.other.comment=fore:#007F7F
colour.embedded.comment=back:#E0EEFF
colour.notused=back:#FF0000

colour.number=fore:#D000FF
colour.keyword=fore:#0055AA
colour.string=fore:#8000FF
colour.char=fore:#008040
colour.operator=fore:#C07040
colour.preproc=fore:#A00070
colour.error=fore:#FFFF00,back:#FF0000

style.stringeol=fore:#888800,back:#FFD0D0,eolfilled,$(font.computer),bold

## Global default styles for all languages
# Default. Set here the back property to change the default background color of Scintilla.
style.*.32=$(font.text),fore:#000000
# Line number
style.*.33=$(font.computer),back:#A8D0C8,fore:#604020
# Brace highlight
style.*.34=fore:#0000FF,back:#AAEEFF
# Brace incomplete highlight
style.*.35=fore:#FF0000,back:#E0C0D0
# Control characters
style.*.36=font:Arial Narrow,size:10
# Indentation guides
style.*.37=fore:#C0C0C0,back:#FFFFFF

Of course, you will want to adjust these settings to your tastes (and installed fonts).

[Update] I give below the expanded version my styles. We loose style relationship ("this style is like this one, but italics") but it will easier to put them in other editors. Again, these are guidelines, not hard rules!

# Expanded version (no references) of ahk.properties
# Can be used in other Scintilla based editors

# Base (background, base font)
style.ahk.32=font:Verdana,size:8
# Default (everything not below: spaces, untyped parameters)
style.ahk.0=font:Georgia,size:9
# Line comment (; syntax)
style.ahk.1=font:Comic Sans MS,size:8,fore:#8080A0
# Block comment (/*...*/ syntax)
style.ahk.2=font:Georgia,size:8,fore:#7070A0
# Escape (`x)
style.ahk.3=font:Georgia,size:9,fore:#8000FF,back:#E0EEFF
# Syntax operator
style.ahk.4=font:Verdana,size:8,fore:#A00070,bold
# Expression operator
style.ahk.5=font:Verdana,size:8,fore:#C07040
# String
style.ahk.6=font:Georgia,size:9,fore:#8000FF
# Number
style.ahk.7=font:Andale Mono,size:8,fore:#D000FF
# Identifier (variable & function call)
style.ahk.8=font:Andale Mono,size:8
# Variable dereferencing %varName%
style.ahk.9=font:Andale Mono,size:8,back:#F0F0FF
# Label, Hotkeys & Hotstrings (& Function definition?)
style.ahk.10=font:Lucida Console,size:9,bold,back:#FFFFE0,eolfilled
# Keyword - Flow of control
style.ahk.11=font:Verdana,size:8,fore:#007F7F,bold
# Keyword - Commands
style.ahk.12=font:Georgia,size:9,fore:#0055AA,bold
# Keyword - Functions
style.ahk.13=font:Andale Mono,size:8,fore:#0055AA
# Keyword - Directives
style.ahk.14=font:Georgia,size:9,fore:#0055AA,back:#E0F0FF,bold
# Keyword - Keys & buttons
style.ahk.15=font:Century Gothic,size:10,fore:#007F7F
# Keyword - Variables
style.ahk.16=font:Andale Mono,size:8,fore:#007F7F
# Keyword - special parameters ("Keywords")
style.ahk.17=font:Andale Mono,size:8,fore:#0055AA,italics
# Keyword - User defined
style.ahk.18=font:Verdana,size:8,fore:#800020,bold
# Variable keyword (built-in) dereferencing %A_xxx%
style.ahk.19=font:Andale Mono,size:8,fore:#007F7F,back:#F0F0FF
# Error (unclosed string, unknown operator, invalid dereferencing, etc.)
style.ahk.20=font:Verdana,size:8,back:#FFC0C0


Known limitations
The biggest limitation is that in AutoHotkey, the nature of parameters of commands depends on the command, the position of the parameter, and even the number of parameters!
For example, in WinMove, we have coordinates that are expressions, titles and texts that can be strings or a special parameter (A) or a special modifier (like ahk_id) followed by a string.
Even worse, the coordinates can be the first parameters or after the first title and text, depending if there are 2 or at least 4 params...
Obviously, this is beyond the capacity of a simple syntax highlighting lexer, otherwise I would have to rewrite the full parser of AutoHotkey! While it would be interesting to detect early syntax errors, this belongs more to a full IDE for AHK.
Scintilla lexers are designed to be flexible and able to handle some context, but by spirit, keyword lists are made external, so they can be easily changed (new commands, etc.). Lexers are compiled code, so modifying them should be scarce, and reserved to those knowing C++ and having a compiler.

Currently, most of these parameters are styled as Default, with highlighting of operators, numbers, etc., even in pure strings: in "What else is left" sentence, I will highlight "else" as flow control keyword and "left" as key name.
In future versions, I want to treat all parameters as strings (the most common case), allowing %var% forms, and respecting % expr syntax.
A possible improvement over this quite rigid policy is to identify some typical keywords (like ahk_group), excluding too common English words like or, normal, low, etc.
And perhaps to apply a rule: if something between two commas, once the left and right spaces are trimmed out, is a valid identifier (one word), highlight it as identifier.
So WinGet result, MinMax, AutoHotkey Help will get the first parameter highlighted as variable, MinMax maybe as "keyword" (special parameter) and AutoHotkey Help as string.
The problem is that this simplistic rule will highlight a single word title as a variable.
So the question I ask right now is: should we have "smarter" rules working most of the time, but sometime giving bad results, or a more general rule, still with bad results (everything is string) but being perhaps more consistent?

Note: the identifier style is currently not used (only internally). It will be used in the version to come.

The next version will correct some issues by adding "back tracking".
Scintilla lexers are designed to be fast by highlighting only the visible lines: when you open a large file, the editor will not parse the whole file, but only the first visible lines. When you scroll down, the newly visible lines will be lexed,but the previously parsed lines will remain "as is". Idem when you modify a line: Scintilla will start lexing from the start of the current line, up to the last visible line. Only when you jump to the end of the file Scintilla will lex the whole source.
Most lexers works by moving forward: they find an operator, they apply the corresponding style; start of comment -> apply until end of line or end of comment; valid start of identifier -> apply identifier style, unless it is in a keyword list, thus change to the style of this list, etc.
Some lexers, for most complex languages like Perl, need "back tracking": they have to go back in the same line, or even some lines before, eg. to see what kind of string quote was used in multiline string.
That's the case for AutoHotkey. For example, I see MsgBox, I highlight it as command, skip some spaces (with default style), then find := Since AHK doesn't forbid to use "keywords" as identifiers, MsgBox here was a simple variable. So I have to go back to change the style of the word to identifier.
Likewise, if I see Foo(bar), I don't know if it is a function call or a function definition (unfortunate design choice...) until I find an opening brace after spaces and comments. (This one, I may skip, ie. hightlight them both as simple identifier).
Again, if I find a line starting with +, I don't know yet if it is a hotkey definition or the continuation of an expression (unless I set a line attribute, perhaps).
So, there are still lot of issues to fix:

Will be addressed soon:

- Esc:: ExitApp -- if followed immediately by LF only (ie. Unix-mode line ending), it is not correctly highlighted.
- When editing a line above an unterminated variable ref, I loose the error style or it leaks.
- A string immediately followed by a % is always lexed as identifier, even if not relevant.
- ( LTrim Join (should get special param style)

Will be addressed when back track will work:

- #PrintScreen:: seen as identifier, so doesn't get label style.
- After a command, parameters must take the string style, except for If (exp) and % exp.
- Set left side of = := etc. to identifier style (except variable dereferencing).

Probably won't be addressed (complex problem of context, rarity of meeting this):

- If the label is starting with some strange legal character (eg. opening parenthesis), the lexer will be fooled (here, it will see the start of a continuation section)...

Plus some unknown issues that maybe you will report after testing this lexer!

For the curious people, you can take a look at the source: LexAHK1.cxx
Note that once these files will be officially submitted to the Scintilla mailing list and integrated to the official distribution, they are likely to be obsolete in this topic!

[Chris]

I know you've worked hard on this. I'm going to try it tomorrow.

One thing that might help is to give an executive summary at the top of your post, which might persuade more people to try it. This is because they wouldn't have to read the whole post (not that it isn't interesting to some of us ). For example, it might be good if you suggested which SciTE-based editor is likely to be the most friendly (for those who have never tried one), as well as where to download it, etc.

Thanks.

[PhiLho]

Yes, I vaguely felt I was spreading too much my URLs in the post, and seeing it again, I found it quite intimidating! :-)
So per your suggestion, I structured a bit my post, thanks.
Happy testing.

[SKAN]

Dear PhiLho,

Great Work! A nice contribution to AHK Community!



I love this Syntax Highlighting thing. But, it took me some time in understanding how to make this stuff work, because :

[*:1shw13g7]I`ve been using only Notepad/Metapad till date. ( I tried to shift to PSPad but could not cope up with it!)
[*:1shw13g7]I`ve never heard the words LEXER & SciTE expect for your references elsewhere in this forum.

Suggestion:

It took me time when it was just 3 simple steps:

- Add a few entries/lines to the file: SciTEGlobal.properties or SciTEUser.properties.
- Copy the file ahk.properties to SciTE folder.
- Take a backup of Scilexer.dll and replace it with your version.

I request you put up a single installation executable (including your Scilexer.dll ) for various Scintilla based editors.
It should ease things for other "Laymen" like me![/list]I need help:

Being a Metapad user I find the interface of Notepad++ (I tried it only today) more friendlier than SciTE.
I am not able to even guess on what I should do to enable Syntax Highlighting for AHK in Notepad++

Please help/advise.

Regards,

[PhiLho]

Dear Goyyah!

Thanks for testing. I thought about putting a screenshot, but I was too lazy! Thanks again for giving one. May I suggest you put a label and/or hotkey/hotstring in your screenshot? Or I should make my own...

I am sorry, I won't make an install, for various reasons:
- I don't like install programs;
- Therefore, I have no experience with install programs (although I took a deep look at NSIS, just in case), and I have no time to spend on them; I can use AHK here, but:
- I guess "various Scintilla based editors" are too different to make easy to built a single install program, and it is time consuming;
- If all goes well, it will become officially part of the Scintilla package.

I don't know Notepad++, but I can try it if it can help you.

[SKAN]

Dear PhiLho,

May I suggest you put a label and/or hotkey/hotstring in your screenshot? Or I should make my own...

It will be nice if you put the Snapshot on the top of this page. Visitors will get an instant idea on what this is about.

I am sorry, I won't make an install, for various reasons:
- I don't like install programs;
- Therefore, I have no experience with install programs (although I took a deep look at NSIS, just in case), and I have no time to spend on them; I can use AHK here, but:
- I guess "various Scintilla based editors" are too different to make easy to built a single install program, and it is time consuming;

No problem. I will write one (Pure AHK!) and post it here. Your stuff is very good and would be a boon to the AHK community. I want it to get noticed and hope many AHK-ites test it so that you receive ample feedback...........

- If all goes well, it will become officially part of the Scintilla package.

.............. before it becomes part of Scintilla package.

I don't know Notepad++, but I can try it if it can help you.

Please do help!. The interface of Notepad++ is definitely friendlier than SciTE.
It does have a SciLexer.dll. but I do not know where/how to put the ahk.properties file!
More AHK-ites would prefer Notepad++, I guess.

Regards,

[SKAN]

Dear PhiLho,

Its okay if I open an existing .ahk file but If I type a new script I do not get the "Syntax Highlighting" and I do not get an extension in the Save File - Save as Type dialog.



Can you please mail me a copy of your SciTEGlobal.properties or SciTEUser.properties?

Regards,

[SanskritFritz]

PhiLho, thanks for your great efforts!
I cannot get it to work under Notepad++ either :-(
I can see in the screenshots there is no folding, as you also wrote. But using the Extras/Editors/Notepad++ stuff, folding is working. Can't you just get the settings from there?

[PhiLho]

@SanskritFritz
Folding is hard-coded like syntax highlighting! Even more, as the keywords are in the code itself.
I downloaded Notepad++, and indeed, it looks like a fine software. I work on the syntax files right now, I will update this message as soon as they are finished. It is the first time I care about other editors. :-)

@Goyyah
You can have my copy of my personal .properties files right on my site (end of home page).
The list of extensions in open.filter is... for the Open File dialog. I never noticed the combo is empty in the Save File dialog! 99% of the time, to create a new AHK file, I just make a copy of an existing file, so the extension and the coloring are already there. But for a new buffer, you can force the choice of language, either by saving the almost empty buffer (no need to choose the extension in the combo, just type it!), or by choosing the language in the... language menu (defined in the menu.language property).

[PhiLho]

Much appreciated. Notepad2 (a SciTE based editor) needs "only" be recompiled to deal with AHK. I would do it myself - unfortunately I don't know C++ at all, nor do I have a compiler :roll:

Ah, that's the price to pay for a lightweight editor. It is much smaller than Notepad++ and compact (no DLL), but it has a smaller choice of supported languages (among the most popular) and upgrading it is much harder.
I think you don't need to know C++ to recompile a project, as long as a build file is provided for the compiler of your choice. If you want to keep the size of Notepad2 small, you should choose the free compiler of Microsoft.
Or ask somebody with a compiler to do this for you, it shouldn't take too long (not me, sorry).

Wouldn't it be of interest to provide a lean/basic editor together with the AHK download (eg. with ahk/ini syntax highlighting only)? Kinda give-away, regardless that once one get used to AHK he/she can swap to Notepad++; UltraEdit, PSPad

In the download, no. Too big, not interesting everyone. Given in the same downlaod page, why not, but that's Chris' choice, and who would start such project? Perhaps using a Notepad2 base: small, a bit rustic (tried it only briefly!) although friendlier than SciTE, open source, so not too hard to adapt.

[SKAN]

Dear PhiLho,

I downloaded Notepad++, and indeed, it looks like a fine software. I work on the syntax files right now, I will update this message as soon as they are finished. It is the first time I care about other editors. :-)

It will be very helpful to the community if you can

[*:2cbyne3o]Recompile Notepad++ to automatically support AHK Syntax Highlighting
[*:2cbyne3o]Provide a download in .ZIP format ( requiring no installation )
[*:2cbyne3o].REG files to associate Notepad++ to be the default editor for AHK
Hope my request is not immoderate. ( This is one nice thing that can happen to AHK & the Community. )

Regards,

[SKAN]

Wouldn't it be of interest to provide a lean/basic editor together with the AHK download (eg. with ahk/ini syntax highlighting only)? Kinda give-away, regardless that once one get used to AHK he/she can swap to Notepad++; UltraEdit, PSPad, etc. ...

Yes! Thats a nice Idea. :DRegards,