Comment by TZubiri
14 hours ago
A lot of criticism of python often mentions the whitespace as lexical scope tokens, and that criticism is usually posited by users of the language.
As implementer of an interpreter, did you feel that whitespace for lexical scoping made the job of writing the lexer significantly more complex?
And, there are multiple white space symbols!
<space><space><tab><space>
is different than
<space><tab><space><space>
So you also have to track the actual sequence of counts of white space used for each level, rather than just a simple count.
Or you just forbid mixing spaces and tabs in the same indentation sequence, the way most whitespace-sensitive languages seem to end up doing. Or you make a slightly more reasonable rule: spaces may follow tabs, but no tabs may follow a space. That's at least unambiguous.
Oh, that's really elegant! I've got a whitespace sensitive language of my own, and I think I'll change it to use that rule! Thanks!
(Until now, I went with the standard approach: Remember the leading whitespace of the previous line. Then compare with the new line's leading whitespace: If they are the same, then no change in indentation. If the old one is a prefix of the new one, it's an indent. If the new one is a prefix of the old one, it's a dedent. If neither, it's an error)
1 reply →
But it also feels arbitrary and annoyingly restrictive. On top of that there are at least 25 whitespace codepoints in UTF. Should your language really be opinionated about when, where, and in what order (for example) the "mongolian vowel separator" appears?
19 replies →
> <space><space><tab><space> is different than <space><tab><space><space>
in my view, both are the same, both `is` (or ===) an IndentationError raise
It's just a stack containing strings at the end of the day. Really not a big deal.
Right, pointers to strings but yeah. Essentially the whitespace count specifies the stack depth at which a line is to be executed. A decrease in stack depth means all superior levels are terminated.
Doesn't affect function call stacks though.
For a 1024 byte implementation (and even way more complex impl.) You would just force one whitespace char, and definitely no mixing.
> did you feel that whitespace for lexical scoping made the job of writing the lexer significantly more complex?
Significant indentation requires a more complex lexer because it means the lexical grammar is no longer regular. The lexer can't just be a finite state machine, instead it has to maintain a stack of previous indentation levels.
But I don't think many modern languages have a regular lexical grammar anyway. Without significant indentation, some other features still require the lexer to maintain a stack - e.g. string interpolation (Python's f-strings).
> that criticism is usually posited by users of the language.
Uhhh, no. Sure, it's posited by people who feel they are are forced to use it, but it's basically unlearning other syntax.
Here's a study about people with no experience. They do better with python:
https://www.researchgate.net/publication/262256894_An_Empiri...
When the scala language made whitespace optional, it was very divisive, but now it's extremely well accepted.
At a former workplace where most stuff was done in PHP, some colleagues used whitespace very liberally. Like, indentation was just a random amount of whitespace, every line slightly different. Sometimes 2 or more spaces between keywords, etc.
After that experience Python code is like eye-bleach to me.
I meant users of languages ( application programmers) as opposed to compiler programmers, not python programmers specifically, so I'm including devs that use other languages and see in python a tool that they would consume.