Comment by redox99
2 days ago
The human genome is 800MB, (and 500MB with compression according to GPT) so that part is easy.
I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".
Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.
Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.
> A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for
What makes it hard to search for? If you tell me you want to open a lemonade stand in East Bumbling Falls, I won't know that city’s municipal ordinances. But I will know that cities tend to have ordinances. So I’d search broadly for what codes that city has. Then I’d go through each one looking for anything that might affect a lemonade stand.
Usually at least here it's tens of thousands of laws like "replace article 3 paragraph 2 with:" and it's not like you can ctrl f "lemonade stand" and get something useful.
A strong llm without that knowledge can probably spend millions of tokens parsing every single ordinance and give you an answer if you directly ask about that.
But both a local human or an "all knowing" llm when you just tell it about the stand will be like "hmm wasn't there a law about low calorie beverages being tax exempt? Let me check" and after a very precise search come back with a tip that if you sell sugar free lemonade that's tax exempt.
I mean would you get an accountant from another country just because he can use search?