Comment by janalsncm
4 hours ago
You are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old.
Basically every part of the original transformer was replaced with something more efficient or better:
LayerNorm -> RMSNorm
Sinusoidal position encoding -> RoPE
MHA -> GQA
ReLU -> GELU
What this means is that there is ample opportunity to improve on what we’ve done thus far.
The original RoPE paper is a great example of the authors not actually understanding how their own discovery was functioning fully which is a fun example of what I’m talking about I guess…
Yes you might progress the field, but will you really understand why? You can make up an explanation and anthropomorphise it with a few contrived diagrams and everyone will cheer!
Oh yeah I absolutely agree that there's lots of greenfield for relative newcomers to find some big improvements! I'm just saying that reasoning about 'best practices' in terms of what works for something like a CPU is missing some important and relevant differences.