Comment by topwalktown
4 months ago
Transformers like Llama use rotary embeddings which are applied in every single attention layer
https://github.com/huggingface/transformers/blob/222505c7e4d...
4 months ago
Transformers like Llama use rotary embeddings which are applied in every single attention layer
https://github.com/huggingface/transformers/blob/222505c7e4d...
Very interesting! Do you know if there were any studies about whether this improves performance?