Comment by evanb
2 months ago
A cartoon:
To form a coherent idea you need to coordinate a lot of tokens. In other words, ideas are long-distance correlations between tokens. Ideas are the long-wavelength features of streams of tokens.
Is it exactly right? No. But as a cartoon it can motivate exploring an idea like this.
Right. This makes sense. But why Fourier space in particular. Why not, for example, a wavelet transform.
> Why not, for example, a wavelet transform.
That is a great idea for a paper. Work on it, write it up and please be sure to put my name down as a co-author ;-)
Or for that matter, a transform that's learned from the data :) A neural net for the transform itself!
3 replies →
Now you’re talking efficiency—-certainly a wavelet transform may also work. But wavelets tend to be more localized than FTs.
This way you end up with time dilated convolutional networks [1].
[1] https://openreview.net/pdf?id=rk8wKk-R-
I like this. Anything that connects new synapses in my skull via analogy is a good post.
This is really a very interesting way of visualizing it.