Comment by aleph_minus_one

3 hours ago

> - Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions.

Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]?

N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid.

I still remember the time when Einstein@Home was in its heyday, and many people who loved putting together fast PCs contributed sometimes even for the reason of showing off in the statistics [3].

---

[1] https://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_f...

[2] https://en.wikipedia.org/wiki/Folding@home

[3] https://einsteinathome.org/de/community/stats

2 comments

aleph_minus_one

Reply
cheriot  3 hours ago

I’ll be impressed if somebody can make that work considering the vastly larger compute required.

  • aleph_minus_one  2 hours ago

    > I’ll be impressed if somebody can make that work considering the vastly larger compute required.

    I think you underestimate the computational ressources that the mentioned (and similar-kinded) scientific projects needed. Also consider how much computational ressources people invested into cryptocurrency mining.

    No, I think the reasons are different:

    - Many companies that train AI model use training data which must not be distributed for copyright reasons (and using it is a legal gray zone)x.

    - Also consider that the amount of training data is insane. Scientific projects (and cryptocurrency mining, too) have the property that typically the amount of data (storage requirements) is small (or at least the computation can be partitioned so that each sub-task needs little data), but the required computing ressources are insane.

    - AI companies consider a huge part of their training data as their "secret sauce" (they often even paid lots of money to generate it, for example by paying world-renowned experts for writing an answer for some important question).

    Thus: Yes, the required computation ressources are huge, but this is a problem for which I consider it to be plausible that it can be solved. The real problems are in my opinion different.