Comment by sashank_1509
5 days ago
If we wanted to make a drone have the intelligence of a dragonfly (which might still be a decade away), we would at least need a billion parameters and a trillion tokens in training data. Just helps us appreciate what nature is capable of.
Sounds about right, 1 million neurons and potentially 1 billion synapses. Back in school I used to think computers and machines were so advanced and complex compared to boring old biology. It was such a silly misconception to be so many orders of magnitude off in complexity.
Tokens? Is that the new « bit »?
I guess we're still not out of the 'flying into a window' stage. https://en.wikipedia.org/wiki/Loitering_munition
Is nature doing that? Or following a much simpler pattern?
Producing complex behavior simply is usually harder and more elegant than using a lot of superfluous complexity.
I feel pretty confident saying it is not following a simpler pattern. It is doing something very hard, with far less training data and a much more alien form of compute (which is also much slower in general than computers).
Is there some elegant learning algorithm in nature that enables this? Maybe. But I don’t think there’s any simplification of the task, that nature has found. For example, I doubt we will find a 100 line algorithm to do the motion the dragonfly did, or a simple physics equation to describe it etc. My intuition, stems from the bitter lesson, a lot of these tasks are irreducibly complex and trying to find elegant simple models, tends to be a waste of time.
Does the how matter if we can’t replicate it the same way?