Comment by seydor
3 months ago
we re going to get closer and closer to removing all hand-engineered features of neural network architecture, and letting a giant all-to-all fully connected network collapse on its own to the appropriate architecture for the data, a true black box.
Which is the Logical conclusion.
If the neural network can distill a model out of complex input data.
Especially when many model are frequently trained through data augmentation practices that actively degrade input to achieve generalisation abilities.
Then why are we stuck wearing silk glove tokenizers?