Comment by chriswarbo

> there's only one syntax for the language(syntax extension and mixed-language documents aside) and it too follows a linear order

I'm of the opinion that programming languages should have two "official" representations: the usual text-based representation, and a machine-readable concrete syntax tree. The latter should be an existing format, e.g. JSON, XML, s-expressions, etc.

It should be "official" in the sense that it's standardised in the same way as the text format, language implementations (compilers/interpreters) should accept it alongside the text format, and they should have a mode which converts the text format into the machine-readable format (and optionally the other way). This is important, since lots of code 'in the wild' only makes sense after feeding it through particular preprocessors, specifically-configured build tools, sed-heavy shell scripts, etc. such that the only tool that can even parse the code correctly is the compiler/interpreter (and even that might need a bunch of tool-specific flags, env vars, config files, etc.!). This makes tooling much harder than it needs to be, and any "unofficial" workarounds will need constant work to keep up with changes to the language.

I say concrete syntax trees since we want to impose as little meaning as possible on the tokens, since that makes tooling more robust in the face of things like macros/custom syntax, new language features, incomplete or malformed code, etc.