Comment by virgildotcodes
4 days ago
> throw stuff at the wall, see what sticks, once something finally does report it as grandiose and claim to be "intelligent".
What do we think humans are doing? I think it’s not unfair to say our minds are constantly trying to assemble the pieces available to them in various ways. Whether we’re actively thinking about a problem or in the background as we go about our day.
Every once in a while the pieces fit together in an interesting way and it feels like inspiration.
The techniques we’ve learned likely influence the strategies we attempt, but beyond all this what else could there be but brute force when it comes to “novel” insights?
If it’s just a matter of following a predefined formula, it’s not intelligence.
If it’s a matter of assembling these formulas and strategies in an interesting way, again what else do we have but brute force?
See what I replied just earlier https://news.ycombinator.com/item?id=47011884 namely the different regimes, within paradigm versus challenging it by going back to first principles. The ability to notice something is off beyond "just" assembling existing pieces, to backtrack within the process when failures get too many and actually understand the relationship is precisely different.
So I don’t really see why this would be a difference in kind. We’re effectively just talking about how high up the stack we’re attempting to brute force solutions, right?
How many people have tried to figure out a new maths, a GUT in physics, a more perfect human language (Esperanto for ex.) or programming language, only to fail in the vast majority of their attempts?
Do we think that anything but the majority of the attempts at a paradigm shift will end in failure?
If the majority end in failure, how is that not the same brute force methodology (brute force doesn’t mean you can’t respond to feedback from your failed experiments or from failures in the prevailing paradigms, I take it to just fundamentally mean trying “new” things with tools and information available to you, with the majority of attempts ending in failure, until something clicks, or doesn’t and you give up).
While I don't think anyone has a plausible theory that goes to this level of detail on how humans actually think, there's still a major difference. I think it's fair to say that if we are doing a brute force search, we are still astonishingly more energy efficient at it than these LLMs. The amount of energy that goes into running an LLM for 12h straight is vastly higher than what it takes for humans to think about similar problems.
at similar quality NN speed is increasing by ~5-10x per year. nothing SOTA is efficient. it's the preview for what will be efficient in 2-3 years
In the research group I am, we have usually try a few approach to each problem, let's say we get a:
Method A) 30% speed reduction and 80% precision decrease
Method B) 50% speed reduction and 5% precision increase
Method C) 740% speed reduction and 1% precision increase
and we only publish B. It's not brute force[1], but throw noodles at the wall, see what sticks, like the GP said. We don't throw spoons[1], but everything that looks like a noodle has a high chance of been thrown. It's a mix of experience[1] and not enough time to try everything.
[1] citation needed :)
I always call it the "Wacky Wallwalker" method (if you're of a certain age, this will make sense to you).
The field of medicine - pharmacology and drug discovery, is an optimized version of that. It works a bit like this:
Instead of brute-forcing with infinite options, reduce the problem space by starting with some hunch about the mechanism. Then the hard part that can take decades: synthesize compounds with the necessary traits to alter the mechanism in a favourable way, while minimizing unintended side-effects.
Then try on a live or lab grown specimen and note effectiveness. Repeat the cycle, and with every success, push to more realistic forms of testing until it reaches human trials.
Many drugs that reach the last stage - human trials - often end up being used for something completely other than what they were designed for! One example of that is minoxidil - designed to regular blood pressure, used for regrowing hair!
It’s almost like the iteration loop refines itself between checks notes in Sutton search and learning