← Back to context

Comment by ftxbro

3 years ago

> Left unsaid in this piece is that OpenAI likely would have to increase parameters

Maybe true, but he also said "We are not here to jerk ourselves off about parameter count"

https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...

Also, who says that the "transformer scaling laws" are the ultimate arbiter of LLM scaling? They overturned previous scaling laws and other scaling laws might overturn them. Furthermore, it's even possible that the transformer model won't even be used in later models. I remember Ilya making the point that just because the transformer model was the first one that looks like it can scale intelligence just by lighting up billions of dollars of GPUs, it doesn't mean it's the last one. Maybe it will even be like, the vacuum tube of AI models, and other ones are being made in secret. A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer.

28 comments

ftxbro

Reply
guyomes  3 years ago

> A hacker news rumor was that they are paying $5M-$20M per year to the top neural net experts probably to make some exotic architectures to surpass transformer

This reminds me a TV interview of the author Patrick Modiano, just after he won the literature Nobel price. The presenter asked him if the money would help. The author answered essentially that the next time he would be in front of a white page, the money surely wouldn't help.

In the case of surpassing transformers, money could help to give access to more compute power. It could also help to prevent the research from being public.

  • ohgodplsno  3 years ago

    Modiano is a rich man, born into a rich family. Wealth doesn't help in front of a white page, but it sure helps being able to stay in front of that white page instead of having to go take up a job because you're not sure what you're eating tonight.

    As always, wealthy people and their "money doesn't make happiness" bullshit.

    • harvey9  3 years ago

      Since he already didn't need to work another job to pay the bills, the extra money from the Nobel prize does not make a difference in this case as he can already put all his time into writing.

  • 0xfffafaCrash  3 years ago

    If someone is already working on a problem full-time, money only helps to the extent that resources they can be buy with money are the limiting constraint. However, beyond deep work needed for a single individual, when you need to explore potential opportunities in a broad space of possibilities, money can hugely effect the search of that space because work needed for major breakthroughs remains parallelizable. You can delegate subtasks to people if you can afford those people. You can hire more of the few specialized people who know about a niche to work on your problem instead of other problems. You can exploit synergies from crosspollination of ideas from bringing together brilliant minds into the same conversations. The influx of money is very very likely to increase the pace of innovation in AI. The breadth of possible avenues for breakthroughs is largely yet-to-be-explored.

tootie  3 years ago

I'm not an expert but isn't size the distinguishing feature of an LLM? It's the first L.

  • hiddencost  3 years ago

    They needed an architecture that could take advantage of the scale, first. That's what BERT did.

jimsimmons  3 years ago

Curious if anyone can confirm $5-20M figure. Seems absurdly high but what do I know

  • hansvm  3 years ago

    Can't confirm OpenAI's position in particular, but $500k/yr/person is table stakes for a decent engineer directly connected to the company's bottom line. Double that for an actual expert, double it again if they're consulting, and put together a team of 3-10 of them. Those numbers aren't too far off.

    • jimsimmons  3 years ago

      I can see $5M per person being possible. $20M is the absurd part. 5 years with such a comp leads one to a net worth that is borderline filthy. Like elite, world renown athletes and actors level of wealth. Again, could all be true but just unexpected from my experience.

      8 replies →

  • afastow  3 years ago

    I wouldn't put any stock into a random twitter rumor by someone likely looking for clout. The source, some guy with likely a purchased checkmark and 12k followers (who knows how few before he claimed to have this insider knowledge), claims four(!) different "extremely reputable" sources that have independently confirmed it. How many people exactly are they making these offers to? Do they all happen to know this guy, someone with no discretion apparently, and everyone decided to tell him this information for what reason exactly?

    99% chance it's made up.

    That said, if they thought a specific individual had even a reasonable chance of coming up with an improvement on the current state-of-the-art AI architecture that they'd be able to keep entirely to themselves, $20M would be a massive bargain.

    The rumor is still almost certainly fake, but for someone very specific at this critical time in the field, I don't know if the number would be that absurd.

    • imtringued  3 years ago

      Twitter rumors also claimed a parameter count of 100 trillion parameters and they visualized it with two circles with a huge size difference to make it look intimidating.

      I guess the reason why AI is so interesting is that human stupidity is so widespread.

    • jimsimmons  3 years ago

      Good point. Super unlikely that the top people all know this guy. There are probably a hundred or so of them in the whole world.

      I’d pay creator of GPT that money easily. Probably not anyone else