I would recommend trying something like PyParsing[1] instead. Libraries like this allow you to compose the parser from language-level entities (object and functions, on top of regex and string literals). This means you can attach comments to those entities naturally within the syntax of the language. You also get much better error reporting out of the box, as well as a well-defined way of attaching transforming code to parts of the parser.
There's a place for simple regexes, but complex regex DSLs (with comments and non-significant whitespace, etc.) are almost always less convenient than simply using your language directly.
I would recommend trying something like PyParsing[1] instead. Libraries like this allow you to compose the parser from language-level entities (object and functions, on top of regex and string literals). This means you can attach comments to those entities naturally within the syntax of the language. You also get much better error reporting out of the box, as well as a well-defined way of attaching transforming code to parts of the parser.
There's a place for simple regexes, but complex regex DSLs (with comments and non-significant whitespace, etc.) are almost always less convenient than simply using your language directly.
[1] https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePypa...