Getting started

Note: this only includes the lexical analyser support by now.

Usage

To run the tool,

python spgen.py -g <grammar-file> -l <language>

Grammar file

A grammar file is used by the tool to describe the syntax of the language you are developing. The rules for the file are simple in general, and this section tries to explain the file format basics.

Lexical analyser

Also known as Lexer. In the lexical analysis a compiler reads and process a stream of data, returning a list of tokens. So something like,

4 - (temp * 2 - 5)

Would return the next list of tokens,

4    : Number
-    : SubstractOperator
(    : LeftParan
temp : Identifier
*    : ProductOperator
2    : Number
-    : SubstractOperator
5    : Number
)    : RightParan

Each token has a type, so 4 is known as Number and the proper value, 4 in this case. These tokens are recognized using rules defined by the grammar file. Each rule on the other hand are defined by a regular expression, which indicate how to identify each token.

Now, for explaining how these tokens are defined nothing works better than an example:

// This file describes an arithmetic processor grammar.

token Number            : '\d'+ ; // One or more digits
token Identifier        : Letter LetterOrDigit* ; // A letter followed by any number of digits or letters
token AddOperator       : '+' ;
token SubstractOperator : '-' ;
token ProductOperator   : '*' ;
token DivideOperator    : '/' ;
token LeftParan         : '(' ;
token RightParan        : ')' ;

fragment LetterOrDigit  : Letter | Digit ;
fragment Digit          : '\d' ;
fragment Letter         : '\w' ;

First of all, each token field corresponds to a Token Rule. It has an id, a grammar and follows the format:

token <id> : <grammar> ;

The <id> indicates the name of the rule, and by convenience, it must follows CapitalizedWords format. The grammar represents a regular expression, similar to those found in other tools. In the rule:

• a string can be described with the text inside of single quotes ('abc'),
• use operators as,
  • *: match any number of the previous rule.
  • +: match the previous rule one or more times.
  • ?: the previous rule is optional.
• use references to other rules just using the name of the other rule. There are limits about this usage however,
• and you can include special inputs in the strings, as '\n' for new lines, '\d' for digits, '\w' for letters, etc.

The rules for the id are defined by the Identifier specification.

Also, a detailed explanation of the grammar can be found in Grammar expression specification.

TODO: include fragments description.