Comment by barrenko
5 days ago
These models / nets / whatever are much "smarter" (loaded term) than we think, we just don't know how to plug-in properly yet.
"We are not interested in the fact that the brain has the consistency of cold porridge." — Alan Turing
I guess Turing couldn’t see the trillions of base pairs of DNA, complex methylation states, dendritic spines of the neurons, etc., just for starters.
This is not how science and engineering work and an arxiv should not be taken at face value.
It has been commonly observed that the current crop of LLMs can be too agreeable/sycophantic (or on some topics, too disagreeable) due to the commonly chosen RLHF priorities.
Simply asking the LLM in two separate contexts the same question but from opposing perspectives, then in a third context asking it to analyze both responses and choose the most neutral and objective take, you wipe out any "(dis)agreeableness" bias and dig closer to a deeper, more nuanced synthesis of a given topic. This paper is just taking this idea to the next level.
This isn't really possible with RLHF alone unless you train the LLM to often give two opposing perspectives, which would get tiring.
Looking at a Problem from various perspectives, even posing ideas, is exactly what reasoning models seem to simulate in their thinking CoT to explore the solution space with optimizations like MCMC etc.
A sufficiently capable AI would be able to plug itself in properly too.
One more reason to be wary of pushing for better capabilities.