Jump to content

Sky Slate Blueberry Blackcurrant Watermelon Strawberry Orange Banana Apple Emerald Chocolate
Photo

ahk lexers and parsers [was syntax highlighting]


  • Please log in to reply
28 replies to this topic

Poll: do you use pygments or geshi for ahk highlighting? (12 member(s) have cast votes)

do you use pygments or geshi for ahk highlighting?

  1. yes (5 votes [41.67%])

    Percentage of vote: 41.67%

  2. no (7 votes [58.33%])

    Percentage of vote: 58.33%

Vote Guests cannot vote
Jamie
  • Members
  • 129 posts
  • Last active: Dec 02 2012 04:59 AM
  • Joined: 26 Mar 2010
Did you abandon the grammar-based parsing in favor of a hand-crafted parser?

I'm looking at creating a grammar for parsing AHK (or a subset of AHK). I've got some simple stuff working, but the unquoted interpolated strings are proving tricky for the lexer because it seems I need the parser to determine when the code should be treated as an interpolated string, and that information needs to direct the lexer. Lex can use start states to change the tokenization rules, and ANTLR can apparently do the same with predicates, but I don't think information can flow backward from parser to lexer without multiple passes.

tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007
you shouldn't have to look more than 1 token ahead to figure out the current state...
autohotkey itself only reads the code once...
I kind of gave up / lost interest for the time being...
take a look at the c++ code in autohotkey.exe or the ironahk code, for how string interpolation is done...

shajul
  • Members
  • 571 posts
  • Last active: Jan 21 2013 08:59 AM
  • Joined: 15 Sep 2006
Thanks for your great lexers.

What can (we|i|you|all_of_us) do to get ahk lexer added to pygments and to github?

I found autohotkey.py and this conversation on google search.

also, is there a way to add this locally to our own git's?

Uberi
  • Moderators
  • 1118 posts
  • Last active: Apr 04 2014 01:20 AM
  • Joined: 23 Aug 2010
tinku99: It's been a while since the last update. I'm trying to get GitHub to highlight AHK code. I've sent a pull request to GitHub, but lexing has been disabled as Pygments has not included your lexer yet. I was hoping you could maybe dredge up the topic again? In the conversation it appears that the thread was left open and forgotten about. Maybe if it was brought up again it would get included?

Excellent work, by the way.

Edit: also tried to ask on #pocoo, but seems pretty quiet there...

fincs
  • Moderators
  • 1662 posts
  • Last active:
  • Joined: 05 May 2007
Please do not forget to include AutoHotkey_L features in the lexers ;)

tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007

tinku99: Pygments has not included your lexer yet. I was hoping you could maybe dredge up the topic again? ...

It wasn't accepted, because there were major issues with it, and it wasn't ready. I have learnt a lot more python since my last attempt, so I will give it another shot... I am restarting from scratch: source. Look at ahklexer.py, it just supports comments and continuation sections for now.
To test it,
1. install pygments : easy_install Pygments
2. git clone <!-- m -->https://github.com/t...9/ahklexers.git<!-- m -->
4. cd ahklexer/pygments
5. python testAhkLexer.py

fincs, I will be working with AutoHotkey_L.

Edit:
Actually it was accepted on 2011-01-03 for pygments 1.4: commit.
But I will submit the new and improved version, once all the comments are in.

tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007
I think the rewritten pygments ahk lexer is about ready.
I ran it on the ahkstdlib lib and demos collected by Tuncay. No errors. sample: richedit.ahk

I'd appreciate any comments / input on making it better.
source

Uberi
  • Moderators
  • 1118 posts
  • Last active: Apr 04 2014 01:20 AM
  • Joined: 23 Aug 2010
Very nice! Didn't break at all on any scripts I tested.

Sent the pull request to enable lexing.

Lexikos
  • Administrators
  • 9397 posts
  • Last active:
  • Joined: 17 Oct 2006
I only noticed two issues:
[*:2h8ejlku]"Static" doesn't appear to be highlighted (as in static variable declarations).
[*:2h8ejlku]For "else return", only "else" is highlighted.

tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007

I only noticed two issues:
[*:351tv0do]"Static" doesn't appear to be highlighted (as in static variable declarations).
[*:351tv0do]For "else return", only "else" is highlighted.

I have added variable declarations to commands.

For now, I am just treating else, return, static, and friends all as commands for now. They only get recognized as commands if they are the first word on a line.
Properly lexing command parameters is going to require more complex logic i think. I will try.

Other changes:
- no longer cheating with (r'.', Text) rule to eat any unmatched characters.
- this means now a few scripts in toralf's stdlib fail on non-ascii characters.
- fixed allowed variable names to include @$#.
- added all ahk escape sequences
- changed labels and hotkeys rules to not match (". As a workaround, you could use +9 or +' as hotkeys if you really want them colored.
- added more easily browsable samples of highlighted scripts here

TODO:
send a pull request to the pygments mercurial repo.

Edit:
pull request 20: updated autohotkey lexer.

tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007
a few cases are still not handled perfectly...
but command parameters are highlighted as strings rather than generic text now.
sample output: <!-- m -->http://golguppe.com/.../pygments/demo/<!-- m -->
source: <!-- m -->https://github.com/tinku99/ahklexers<!-- m -->

maul.esel
  • Members
  • 790 posts
  • Last active: Jan 05 2013 09:26 PM
  • Joined: 28 Feb 2011
Thanks for this lexer!

In pygments v1.4, if is highlighted as a function:
source:
if (A_Username = "a")
	MsgBox a
Output:
[color=red]<span class='nf'>if</span>[/color]<span class='w'> </span><span class='p'>(</span><span class='nb'>A_Username</span><span class='w'> </span><span class='o'>=</span><span class='w'> </span><span class='s'>"a"</span><span class='p'>)</span>
<span class='w'>	</span><span class='k'>MsgBox</span><span class='w'> </span>a
Sorry if this has already been fixed.

Regards
maul.esel

Edit:
Also I'm wondering why the apostroph is highlighted as error?
MsgBox He's very clever.
<span class='k'>MsgBox</span><span class='w'> </span>He[color=red]<span class='err'>'</span>[/color]s<span class='w'> </span>very<span class='w'> </span>clever<span class='p'>.</span>


tinku99
  • Members
  • 560 posts
  • Last active: Nov 18 2013 02:43 AM
  • Joined: 03 Aug 2007

if is highlighted as a function:...
why the apostroph is highlighted as error?

for now, i am just treating "if", "for" and friends as functions as opposed to commands as they expect expressions. I could change them to highlight as builtins, but was defering to a full parser. I have been able to feed the pygments lexer output to both PLY and antlr, but hadn't done beyond a proof of concept. <!-- m -->https://github.com/t.../antlr-pygments<!-- m -->
<!-- m -->https://github.com/t... ... tAhkPly.py<!-- m -->

the apostrophe error is probably from "He'svery clever" not correctly parsing as a string after a command. Probably need to stop treating the apostrophe as a punctuation or something...

maul.esel
  • Members
  • 790 posts
  • Last active: Jan 05 2013 09:26 PM
  • Joined: 28 Feb 2011
Hi again, I discovered a new issue:
Single-line commands are enclosed by
<span class="[color=red]c-Singleline[/color]">	; resolve type[color=blue]</span>[/color]
This causes them not to be highlighted properly on github. Github support says that's an error in pygments and this class should not be there.