Comment by nattaylor
3 months ago
Is pre-training in FP8 new?
Also, 10M input token context is insane!
EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B is BF16 so yes, it seems training in FP8 is new.
3 months ago
Is pre-training in FP8 new?
Also, 10M input token context is insane!
EDIT: https://huggingface.co/meta-llama/Llama-3.1-405B is BF16 so yes, it seems training in FP8 is new.
Deepseek v3 was FP8