That's still an algorithm that underwent immense amounts of tuning. The tuning here on gzip is very simple, very few parameters, and generic. There is no reason to reject it.
Also I don't know about calling GPT-2 "really tiny". You can get coherent responses out of 5-10M parameters.
Heck, a few weeks ago someone posted a 25K-ish generator running on an 8-bit machine (ZX Spectrum? BBC Micro?). It emitted semi-plausible random English phrases. So tiny is subjective but I still think GPT-2 qualifies as a tiny model by any modern standard. I could run it on a laptop.
No--even a really tiny, underpowered model like GPT-2 with no system prompt produces coherent (though not necessarily desirable or correct) responses.
That's still an algorithm that underwent immense amounts of tuning. The tuning here on gzip is very simple, very few parameters, and generic. There is no reason to reject it.
Also I don't know about calling GPT-2 "really tiny". You can get coherent responses out of 5-10M parameters.
Heck, a few weeks ago someone posted a 25K-ish generator running on an 8-bit machine (ZX Spectrum? BBC Micro?). It emitted semi-plausible random English phrases. So tiny is subjective but I still think GPT-2 qualifies as a tiny model by any modern standard. I could run it on a laptop.
1 reply →