funcparserlib

1.0.1 · active · verified Thu Apr 16

funcparserlib is a recursive descent parsing library for Python, based on functional combinators. It is currently at version 1.0.1 and is primarily designed for parsing small languages or external Domain Specific Languages (DSLs). The library provides a concise and type-hinted API, focusing on ease of parser construction without extensive manual lookahead management. It maintains an active development status with a moderately paced release cadence.

Common errors

```
SyntaxError: multiple exception types must be parenthesized
```
cause Using an older version of funcparserlib (e.g., 0.3.6) with modern Python versions (e.g., Python 3.13) which no longer support Python 2-style exception syntax (e.g., `except NoParseError, e:`).

fix Upgrade `funcparserlib` to version 1.0.0 or newer, which has modernized its codebase for Python 3 compatibility. `pip install --upgrade funcparserlib`.
```
funcparserlib.parser.NoParseError: got unexpected token: X
```
cause In older versions of funcparserlib, parse error messages were less informative, only showing the unexpected token without indicating what was expected.

fix Upgrade `funcparserlib` to version 1.0.0 or newer. Version 1.0.0 improved parse exceptions to include expected tokens and grammar rules at the stopped position, making debugging easier.
```
TypeError: 'str' object is not callable (when using `>>` with an unexpected argument)
```
cause The `>>` operator expects a function as its right-hand operand to transform the parser's result. If a non-callable object (like a string) is provided, this error occurs.

fix Ensure the right-hand side of `>>` is a callable (e.g., a function, a lambda, or a type constructor) that accepts the output of the left-hand parser and returns the desired transformed value.

Warnings

breaking Future versions (e.g., 2.0.0 and beyond) will drop support for Python 2.7. If you rely on Python 2.7, ensure you pin your `funcparserlib` version to `<2.0.0` (e.g., `~=1.0`).
Fix: Upgrade your Python environment to 3.8+ or pin `funcparserlib` to a 1.x version if Python 2.7 compatibility is required.
breaking Version 1.0.0 dropped support for Python versions 3.4, 3.5, and 3.6. Using `funcparserlib` 1.0.0+ with these Python versions will lead to incompatibilities.
Fix: Upgrade your Python environment to 3.7+ (or 3.8+ for full current support) to use `funcparserlib` 1.0.0 or newer.
gotcha The `Parser.__init__()` constructor is considered internal and may change in future versions. Directly instantiating `Parser` is discouraged.
Fix: Always use primitive parsers like `tok()`, `a()`, `some()`, `forward_decl()`, `finished`, and parsing combinators (`+`, `|`, `>>`, `many()`, etc.) to construct new parser objects.
gotcha When combining parsers with the `+` operator (e.g., `p1 + p2`), the result can be a tuple of parsed values. If any component parser is skipped (e.g., `-p` or `skip(p)`), the tuple structure might be affected, potentially leading to unexpected results if not explicitly handled.
Fix: Explicitly transform the parsing result into the desired structure using the `>>` operator with a lambda function (e.g., `(p1 + p2) >> (lambda x: MyNode(x[0], x[1]))`) or use `-p` / `skip(p)` to prevent unwanted elements from appearing in the result tuple.

Install

pip install funcparserlib Install latest stable version

Imports

make_tokenizer

from funcparserlib.lexer import make_tokenizer

TokenSpec

from funcparserlib.lexer import TokenSpec

Token
```
from funcparserlib.lexer import Token
```
tok
```
from funcparserlib.parser import tok
```
Parser
```
from funcparserlib.parser import Parser
```
many
```
from funcparserlib.parser import many
```

forward_decl

from funcparserlib.parser import forward_decl

finished

from funcparserlib.parser import finished

Quickstart

This quickstart demonstrates how to define a tokenizer using `make_tokenizer` and then build a simple parser for 'number operator number' expressions using `funcparserlib`'s combinators like `tok`, `+`, and `>>`. It covers tokenizing input, defining terminal parsers, and combining them into a full grammar, finally parsing a sample string.

from typing import List
from funcparserlib.lexer import make_tokenizer, TokenSpec, Token
from funcparserlib.parser import tok, Parser, many, finished

def tokenize(s: str) -> List[Token]:
    specs = [
        TokenSpec('whitespace', r'\s+'),
        TokenSpec('number', r'\d+'),
        TokenSpec('op', r'[+-*/()]'),
    ]
    tokenizer = make_tokenizer(specs)
    return [t for t in tokenizer(s) if t.type != 'whitespace']

# Define parsers
number_parser: Parser[Token, int] = tok('number') >> (lambda t: int(t.value))
op_parser: Parser[Token, str] = tok('op') >> (lambda t: t.value)

# A simple grammar for 'number op number'
expr_parser: Parser[Token, tuple] = number_parser + op_parser + number_parser

# Parse a string
input_str = "123 + 45"
tokens = tokenize(input_str)
result = (expr_parser + -finished).parse(tokens)

print(f"Input: {input_str}")
print(f"Tokens: {tokens}")
print(f"Parsed result: {result}")
# Expected: (123, '+', 45)

view raw JSON →