Comment by LarsDu88
4 days ago
Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.
Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
The 'Exploring Image Models' section addresses that specifically, particularly the 'What Neurons Do Under Pressure' subsection.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
> far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth … your standing in the ocean or not based on your coordinates
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
Islands?
…can be included in a shape file while it remains “small”
All functional knowledge is compression and contextual decompression. Sometimes this compression can be lossless or nearly so. Compression doesn't automatically mean loss of information.
where's this specific example/experiment from? curious to read more, never heard of this
You ask a LLM "Land or Water: lat, long". Do it for enough points and you get a map of the world:
https://x.com/celestepoasts/status/2103232383139057950
Of course this doesn't mean the model has memorized all the lat, long points. Instead it means the model has a representation that allows it to work out if a lat, long is land or water in a analogous way to how a human takes a lat, long and looks it up on a map (a literal "world model" if you like).
It's a great demonstration of how LLMs aren't "Stochastic parrots" but instead have complex representations.