Comment by nullc
2 days ago
Might be useful to use gabor filters as the basis function, since just 2d cosine filters doesn't produce particularly sparse output for angled features. The additional overcompleteness would probably be helpful for the NN learning.
A fun little bit of trivia: Mammalian brains implement Gabor filters in the primary visual cortex (V1), as the first step of the visual processing pipeline.