Comment by jcelerier

I don't think this is optimizing for the right thing, I've sat in front of hundreds of gcc & clang compiler time traces and lexing is a minuscule percentage of the time spent in the compiler