Comment by cschmidt
2 hours ago
Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.
And screw all the 0.1% haters on here, this is great stuff.
That is my reaction too. It looks like great work!
Valuable not only for inference, but for training too (think proprietary datasets).
I would add, a single individual did this.
One person can make a difference :-)
[flagged]
I’m not sure why you think this is ai slop. I work on tokenization research full time. My name is Craig Schmidt and I have a number of papers in the field. This researcher has done some very impressive work and I’m trying to defend him from the HN dismissive hoards.
There is a serious research community on tokenization, and we are quite interested in this work.
Which part of “doesn’t seem to be aislop signs” made you think I believe your comment was aislop?
As I said, It just might be the low information density approach to talking recently that seems off to me.
To explain a bit more, I kept reading and waiting for the penny drop but nothing.
“I cloned your repo” — okay then what happened? Nothing? I put my shoes on this morning.
“I know the tokrniziation community wants to absord tje lessons” okay? Tell me the point please! I know the AI community wants to understand the universe.
“These are useful ideas”. Yes.
Of course, you could argue that my comment falls in the same category in the sense that I am not actually contributing to the topic at hand but I am peeved with all low the information noise.