Comment by maxbond
2 days ago
I've been dabbling in using Fourier analysis in deep learning lately, and I'm surprised it that I haven't turned up very much research in this area (Fourier Neural Operators being what seems to be the biggest exception). Fourier analysis is such a ubiquitous tool, intuitively I'd think it would work great for deep learning. My suspicion has been that complex numbers are difficult to work with, and maybe I'm just bad at surfacing the relevant research, but I'd be interested to hear from those better informed. (My naive approach has been to simply concatenate the real and complex components together into an n+1 dimensional tensor, but surely there's a way that better respects the structure of complex numbers.)
Limited intuitive interpretability of phase likely restricts the broader use of discrete Fourier transforms in machine learning. Frequency, time, and amplitude are tangible and intuitive concepts, whereas phase often feels awkward and less accessible. Using a power spectrum is common practice, but it comes at the cost of losing precision.
RoPE is somewhat related, I think, and it's pretty popular.
There's also 2D rope for ViT, but I don't know how it works exactly.
Convolutional neural networks are pretty big