Comment by PaulHoule

4 years ago

Ashby strikes again.

Current sequence models don't have the right structures to represent math. Even if they use floating point internally, they can't really float the point because the nonlinearity in the model has a certain scale.

A system that processes language can take advantage of the human desire for closure

https://www.eurogamer.net/blood-in-the-gutter

to fool people into thinking it is more capable than it really is. Math isn't like that.

40 comments

PaulHoule

Reply
lupire  4 years ago

Floating point isn't relevant here.

The problem is that human language is approximate and correct math is not, so pattern matching on prose text is doomed. AI trained on exact math does a lot better. But that's not fully generic so fails the weird GPT goal of modeling all of human intelligence through prose. That's not how people solve math at all.

GPT's "Superficially plausible but wrong" math is actually pretty good match for non-expert bad-at-math average human behavior.

hey_over_here  4 years ago

Mwell, the article claims, and points to work that also claims, that large language models can actually be made to perform arithmetic well. They need fine-tuning, verification, chain of thought prompting and majority voting to be combined but the linked Google blog says that Minerva hit 78.5% accuracy (on the GSM8K benchmark).

For me the problem is that we can look at the output and say if it's right or wrong, but we know what language models do, internally: they predict the next token in a sequence. And we know that this is no way to do arithmetic, in the long run, even though it might well work over finite domains.

Which is to say, I'm just as skeptical as you are, and probably even more, but I think it's useful to separate the claim from what has actually been demonstrated. Google claims its Minerva model is "solving maths problems" but what it's really doing is predicting solutions to problems like the ones it's been fine-tuned on, and those problems are problems stated at least partly in natural language, not "naked" arithmetic operations. In the latter, language models are still crap because they can't use the context of the natural language problem statement to help them predict the solution.

Btw, "chain of thought prompting" if I remember correctly is a process by which an experimenter prompts the language model with a sequence of intermediary problems. So it's not so much the model's chain of thought, as the experimenter's chain of thought and the experimenter is asking the model to help him or her complete their chain of thought. I have a fuzzy recollection of that though.

sharemywin  4 years ago

computers already do math. language models just need to translate problems into code of some kind that can be run to get the answer.

executive function/planning is probably the biggest problem at this point for ai.

  • zackmorris  4 years ago

    That's interesting, I hadn't made the connection between executive function and intelligence.

    I went through a burnout in 2019 that felt like having a stroke. My brain finally reached such a level of negative reinforcement after years of failure that it wouldn't let me work anymore. I'd go to do very simple tasks, everything from brushing my teath to writing a TODO list, and it was like the part of my brain that performed those tasks wasn't there anymore. Or at least, it no longer obeyed if it perceived a potential reward involved. It was like my motivation got reversed. I had to relearn how to do everything, despite knowing that no reward might come for a very long time, which took at least 6 months before I began recovering. The closest answer I have is that my brain healed through faith.

    I only bring it up because executive function may be associated with a subjective experience of meaning. If there's truly no point to anything, then it's hard to summon the motivation to string together a sequence of AI tasks into something more like AGI.

    I guess that's another way of saying that nihilism could be the final hurdle for AGI to overcome. It's like the human philosophical question of why there's something instead of nothing. Or why angels would choose to be incarnate on Earth to experience a life of suffering when it's so much easier to remain dissociated.

  • sharemywin  4 years ago

    The point I'm trying to make is LLMs don't need to do everything just be the glue to other systems.

    • enord  4 years ago

      Wait what? Glue as in extract high level semantic representations from _syntatic probabilities_ and pass on to appropriate domain specific tools?

      This is the glaring hole in LLMs, a paradoxical semantic incoherence despite impressive sentenial and gramatical coherence.

      As glue it is so thin as to be potable.

      5 replies →

  • the_af  4 years ago

    > language models just need to translate problems into code of some kind that can be run to get the answer

    A huge "just"! Isn't this the magic step? Translating ambiguous symbols to meaning and combining them in meaningful ways is a big deal which, apparently, these AI models cannot do. They can just parrot things.

    • nl  4 years ago

      > Translating ambiguous symbols to meaning and combining them in meaningful ways is a big deal which, apparently, these AI models cannot do.

      Plenty of AI models do exactly this. Very clear examples include question answering models and code generation. In both cases novel, meaningful responses are generated.

      > They can just parrot things.

      That isn't true. While language models can parrot things it is generally special conditions that make them do it. Specifically, the conditional probability of the next character (or BPE or word depending on the model) has to be much higher than anything else which happens when the thing being parroted is unique text.

      If you ask most Americans or a language model what word comes next in this: fourscore and seven year.. they'll give the same answer, for the same reason.

      6 replies →

  • JacobiX  4 years ago

    I'm not so sure about that. Of course computers can do arithmetic operations, but this is not the same as solving math problems, proving theorems, etc. Even mathematical objects are approximated up to an approximation error in a computer (like a differentiable manifold or a real number).

    • thwayunion  4 years ago

      > Of course computers can do arithmetic operations, but this is not the same as solving math problems, proving theorems, etc.

      Computers can solve math problems and prove theorems; this remains a significant subfield of Computer Science with lots of industrial use cases. However, pure machine learning based approaches toward these problems remain subpar.

      > Even mathematical objects are approximated up to an approximation error in a computer (like a differentiable manifold or a real number).

      Only because it caught on (and in the case of non-computationally-intensive applications, for purely historical reasons). For example, Mathematica has Reals and even functionality for Reals that is literally impossible to implement for integers [1,2]. There are also precise characterizations of objects in differential geometry [3]. You could imagine applying LLMs to these types of programs a la Copilot, but when you do this you will find yourself agreeing with Paul Houle's observation that math is harder to fake than eg art, language, or even glue code for web apps.

      [1] https://reference.wolfram.com/language/ref/Reduce.html

      [2] https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...

      [3] https://github.com/bollu/diffgeo

      1 reply →

    • PaulHoule  4 years ago

      There has been big progress in automated theorem proving lately

      https://en.wikipedia.org/wiki/Automated_theorem_proving

      you just don't hear about it much because the technology is not so fashionable today. Also it is more clear what the limits are, I mean, Turing, Godel, Tarski and all of those apply to neural networks as well any other formal system but people mostly forget it.

      Knuth wrote a really fun volume of The Art of Computer Programming about advances in SAT solvers which are the foundation for theorem provers

      https://www.amazon.com/Art-Computer-Programming-Fascicle-Sat...

      Everybody is aware that neural network techniques have improved drastically in performance, it's much more obscure that the toolbox of symbolic A.I. has improved greatly. Back in the 1980s production rules engines struggled to handle 10,000 rules, now Drools can handle 1,000,000+ rules with no problems.

      3 replies →

    • IshKebab  4 years ago

      Computers are much much better at all that stuff than almost everyone too. Try asking Wolfram Alpha to solve something. Computers have gotten really good at proving things in the last couple of decades and formal verification methods are becoming increasingly popular.

      I think sharemywin is probably on to something. It's going to be really hard for an AI to prove that e.g. x>0 && x+y <= 1 && y>1 is unsatisfiable, but it's trivial for an SMT solver. On the other hand it probably isn't that much of a leap to make an AI that can feed that problem into an SMT solver.

    • Sharlin  4 years ago

      Well, you don't need anything else than basic arithmetic to encode the entirety of, say, ZFC, enumerate every proposition in it, and halt iff you find a proof of whatever theorem you're after. It just might take a while…