Comment by BoppreH

If you're on Python and your problem is too big for a single regex, but too small for a full-blown parser like in the article, I can recommend the built-in re.Scanner[0]. It's a regex-based tokenizer[1] (so no recursive parsing) in the `re` module, that for reasons is completely missing from any official documentation.

You give it a pattern for each token type, and a function to process each match, and you get back a list of processed tokens.

Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive `re.findall` with capture groups will ignore unmatched characters. You also get a reference to the running scanner, so you can record the location of the match for reporting errors.

    import re

    # Initialize the scanner.
    scanner = re.Scanner([
      (r"\d+",   lambda scanner, token: ('COUNT', int(token))),
      (r"\w+",   lambda scanner, token: ('ANIMAL', token.removesuffix('s'))),
      (r"[,. ]",     None), # None -> skip token.
    ])

    # Scan a sample string.
    results, remainder = scanner.scan("45 pigeons, 23 cows, 11 spiders.")

    # Ensure we parsed the whole thing.
    assert not remainder

    print(results)
    # [('COUNT', 45),
    #  ('ANIMAL', 'pigeon'),
    #  ('COUNT', 23),
    #  ('ANIMAL', 'cow'),
    #  ('COUNT', 11),
    #  ('ANIMAL', 'spider')]

[0]: https://stackoverflow.com/a/693818/252218

[1]: https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization