← Back to context Comment by lkbm 1 day ago Yes, but doesn't the token change mean that? 3 comments lkbm Reply clickety_clack 1 day ago You can train a tokenizer on old data just like you can train a model on old data. wongarsu 1 day ago But you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch dannyw 16 hours ago A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.
clickety_clack 1 day ago You can train a tokenizer on old data just like you can train a model on old data. wongarsu 1 day ago But you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch dannyw 16 hours ago A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.
wongarsu 1 day ago But you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch dannyw 16 hours ago A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.
dannyw 16 hours ago A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.
You can train a tokenizer on old data just like you can train a model on old data.
But you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch
A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.