← Back to context

Comment by bgirard

2 months ago

It's really neat that the prompt was released!

I'm curious how many unsolved problems are tried against frontier models when they come out. Are we trying every problems against every release? What is the solve success rate? Is there a sub-community within Mathematics that is coordinating this effort? How much untapped opportunity is there here?

19 comments

bgirard

Reply
emil-lp  2 months ago

The prompt was released, but not the cost of the result.

  • riknos314  2 months ago

    Assuming all 64 subagents were running for a full hour (the tweet states just under an hour):

      Throughput                    Output tokens   Output cost
  ----------------------------  -------------   -----------
  40 tok/s  (5.5 low)                   ~9.2M         ~$275
  55 tok/s  (5.5 base)                 ~12.7M         ~$380
  70 tok/s  (5.5 high)                 ~16.1M         ~$485
  750 tok/s (Sol Fast, $75/M)         ~172.8M       ~$13,000

    Claude estimates that tool use / input tokens might add 10-15% on top of that depending on exactly how the model went about the task.

    Edit: better tok/s estimate buckets based on GPT 5.5 actual speeds since I couldn't find real benchmarks on 5.6 published anywhere. Also account for Sol Fast pricing.

    • conradkay  2 months ago

      Sol fast isn't the Cerebras 750 tok/s version, it's just 1.5x speed at 2.5x price

      I assume they didn't use the Cerebras version for this since it's probably very supply-constrained right now

      4 replies →

  • therobots927  2 months ago

    And not how many times it was prompted before it returned a working solution.

    Or how many prior variants of this prompt were tried.

    Or if proof checking software was used to hone in on the final winning prompt / LLM output.

edflsafoiewq  2 months ago

I find it kind of interesting the whole output wasn't released. A common criticism of mathematical writing is results are "pulled out of a hat"; you only write up a polished, final proof, but hide everything that went into developing it. It's kind of ironic the practice is even carried on when an LLM writes the proof.

not-a-llm  2 months ago

pretty sure already millions of dollars (in inference costs) were already thrown at the Riehmann hypothesis

as the models get stronger, larger amounts will be thrown at it

imagine paying "just $1 bil" to go down in history as the company who's model solved the hardest/most famous open problem in mathematics. imagine the worldwide press headlines.

as they say, the Riehmann Hypothesis is the hardest way to earn a million dollar

  • CSMastermind  2 months ago

    I mean if there's something I'd bet against being solved by LLMs in my lifetime it's that one. We truly do not have line of sight into what a proof would even look like.

    • blovescoffee  2 months ago

      Why would you bet against it being solved by LLMs? Isn't this very post proof that LLMs in an agentic harness are capable of doing real math? If you just keep cranking away at the tokens I don't see a principled argument against that leading to more solutions to unsolved math, even the hardest problems.

      2 replies →