Tokens & AST
The language is first lexed into tokens, out of which an abstract syntax tree (AST) is parsed.
Tokens
The implementation of the lexer can be found at compiler\dd-lexer.
These are valid tokens in the language:
| Name | Pattern | Example |
|---|---|---|
| Whitespace (skipped) | r"[ \t\r\n]+" | |
| Comments (skipped) | r"//[^\n]*" | // ... |
| DocCommentLine | r"///[^\n]*" | /// ... |
| Ident | r"\p{XID_Start}[\p{XID_Continue}-]*" | Foo-bar |
| CurlyOpen | { | |
| CurlyClose | } | |
| BracketOpen | [ | |
| BracketClose | ] | |
| Comma | , | |
| Colon | : | |
| Underscore | _ | |
| Arrow | -> | |
| Star | * | |
| Try | try | |
| As | as | |
| Allow | allow | |
| Default | default | |
| CatchAll | catch-all | |
| Stride | stride | |
| Num | r"-?[0-9][_0-9]*" (decimal) | 01_23 |
| Num | r"-?0b[_0-1]+" (binary) | 0b11_00 |
| Num | r"-?0o[_0-7]+" (octal) | 0o01_23 |
| Num | r"-?0x[_0-9a-fA-F]+" (hexadecimal) | 0xAA_bb |
| Access | RW / RO / WO | |
| ByteOrder | BE / LE | |
| BaseType | uint / int / bool | |
| Integer | u8 / u16 / u32 / u64 / i8 / i16 / i32 / i64 | |
| AddressMode | mapped / indexed | |
| String | r#""[^"]*""# | "my string" |
The tokens are lexed using logos. The regexes are processed by the Rust Regex crate.
Direct tokens have priority over regexed tokens.
Abstract syntax tree
The implementation of the parser can be found at compiler\dd-parser.
The tokens are parsed through multiple sub-parsers into nodes. The AST is one node acting as the root.
The railroad diagrams and ebnf are generated by chumsky and is known to not be 100% correct/complete. Contributions there are encouraged!
The parsed numbers are parsed into a specific type of integer which is displayed in the diagrams. Their sizes are mostly implementation details, with the exception of numbers parsed as bytes.
Node
def_1 ::= ((((({ DocCommentLine } Ident) Ident|Underscore) [ repeat ]) { simple-expression }) [ type-specifier ]) [ node-body ];
def_1
Examples:
register Foo {
address: 0,
}
field Foo 7:0 RW -> _
Specific node types will have restrictions on what is and is not allowed or required. More about that can be found in the reference chapters for those node types as that’s part of the MIR and not the AST.
Repeat
(BracketOpen ((Num<NonZero<u32>>
| Ident) (Stride Num<i32>))) BracketClose
Examples:
[4 stride 2]
[Foo stride 2]
Simple-expression
range
| BaseType
| Integer
| Num<i128>
| Default (Num<i128>
| Underscore)
| CatchAll (Num<i128>
| Underscore)
| byte-array
| Allow
| Access
| ByteOrder
| Underscore
| String
| AddressMode
Type-specifier
(Arrow (BaseType
| Integer
| Underscore)) [ (As [ Try ]) (node
| Ident) ]
Examples:
-> u8 as try Foo
-> bool
-> _ as enum Foo { }
Node-body
(CurlyOpen [ (({ property [Comma]} Comma) { node [Comma]}
| { property [Comma]}
| { node [Comma]}) [ Comma ] ]) CurlyClose
Example:
{
property: _,
register Node {
},
}
Property
{ DocCommentLine } (Ident (Colon (simple-expression
| node
| Ident)))
Examples:
/// Docs
prop1: Foo
prop2: 7:0
Range
(Num<i128> Colon) Num<i128>
Example:
7:0
Byte-array
(BracketOpen ({ Num<u8> [Comma]} [ Comma ])) BracketClose
Example:
[0, 1, 2, 3, 4]