You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.