← Back to context

Comment by domnomnom

4 years ago

Can you ELICSUndergraduate. Tokenizing is normally for if you're writing a compiler of DSL right?

Yes. But any data format you read, particularly any plaintext data format you read, is essentially interpreting or compiling a DSL. On a typical job, people are writing compilers much more often than they think!

It is a general term for the process of breaking a string into "tokens" which have a sort of meaning. Definitely a common task in compilers, but not limited to it.