Comments on: On Lexers, Syntax Highlighting and a good suggestion. http://mikegrouchy.com/on-lexers-syntax-highlighting-and-a-good-suggestion/ Thu, 21 Jan 2010 21:20:37 -0700 http://wordpress.org/?v=2.9.1 hourly 1 By: Mike http://mikegrouchy.com/on-lexers-syntax-highlighting-and-a-good-suggestion/comment-page-1/#comment-27 Mike Sat, 18 Jul 2009 12:18:19 +0000 http://mikegrouchy.com/?p=100#comment-27 I have actually checked out pygments and it did look good, Ideally I'm in no rush to get the project done(and don't know what I am going to use it for) so at this point, I am pretty sure I am going to code everything myself. As for using FSM(or FSA I guess), I have considering that as well, being able to only read each character in the text once would mean the it will be fast, and fast and small are what I am going for. So that is definitely an option I am considering. I have actually checked out pygments and it did look good, Ideally I’m in no rush to get the project done(and don’t know what I am going to use it for) so at this point, I am pretty sure I am going to code everything myself.

As for using FSM(or FSA I guess), I have considering that as well, being able to only read each character in the text once would mean the it will be fast, and fast and small are what I am going for. So that is definitely an option I am considering.

]]> By: Jason G http://mikegrouchy.com/on-lexers-syntax-highlighting-and-a-good-suggestion/comment-page-1/#comment-26 Jason G Fri, 17 Jul 2009 20:27:56 +0000 http://mikegrouchy.com/?p=100#comment-26 You can write a trivial lexer in no time at all, but it'll most likely be relatively inefficient. By trivial I'm thinking one that tries all possibilities from the current character. Rather brute force. One suggestion is to write up your own lexer-like "generator" based off of some format. That way you can reuse the code over all the languages you support. For example, http://en.wikipedia.org/wiki/Flex_lexical_analyser is something I've used in the past, and you could use similar ideas. Also, if you're open to using existing software, I have another suggestion. Since you're using Python/Django, http://pygments.org/ would be an option for lexers. If not, even checking out its source might give you some ideas. Finally, if you're just lookin' for some learnin', then implementing one yourself is good stuff. If you'd really like to get in all kinds of learning, consider building a (finite) state machine from something similar to the input that Flex uses. Have fun! :D P.S. Thanks for the compliments. You're too kind :P You can write a trivial lexer in no time at all, but it’ll most likely be relatively inefficient. By trivial I’m thinking one that tries all possibilities from the current character. Rather brute force.

One suggestion is to write up your own lexer-like “generator” based off of some format. That way you can reuse the code over all the languages you support. For example, http://en.wikipedia.org/wiki/Flex_lexical_analyser is something I’ve used in the past, and you could use similar ideas.

Also, if you’re open to using existing software, I have another suggestion. Since you’re using Python/Django, http://pygments.org/ would be an option for lexers. If not, even checking out its source might give you some ideas.

Finally, if you’re just lookin’ for some learnin’, then implementing one yourself is good stuff. If you’d really like to get in all kinds of learning, consider building a (finite) state machine from something similar to the input that Flex uses.

Have fun! :D

P.S. Thanks for the compliments. You’re too kind :P

]]>