Comment by stymaar

That could be. Just use pre-training for language understanding and let the post-training on synthetic data do the heavy lifting.