Comment by beering

The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.