Comment by NewsaHackO

3 days ago

So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?

17 comments

NewsaHackO

Reply
narush  3 days ago

Our largest funding was through The Audacious Project -- you can see an announcement here: https://metr.org/blog/2024-10-09-new-support-through-the-aud...

Per our website, “To date, April 2025, we have not accepted compensation from AI companies for the evaluations we have conducted.” You can check out the footnote on this page: https://metr.org/donate

  • iLoveOncall  3 days ago

    This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about).

    Not all payment is cash. Compute credits is still by all means compensation.

    • golly_ned  3 days ago

      Those are compute credits that are directly spent on the experiment itself. It's no more "compensation" than a chemistry researcher being "compensated" with test tubes.

      1 reply →

    • rsynnott  2 days ago

      While it would be an ethical concern if they _hadn't_ disclosed it, it's not compensation; it was used _as part of the study_.

    • gtsop  3 days ago

      Are you willing to be compensated with compute credits for your job?

      Such companies spit out "credits" all over the place in order to gain traction and enstablish themselves. I remember when cloud providers gave vps credits to startups like they were peanuts. To me, it really means absolutelly nothing.

      2 replies →

resource_waste  3 days ago

>which is not in any academic journal, or has no peer reviews?

As a philosopher who is into epistemology and ontology, I find this to be as abhorrent as religion.

'Science' doesn't matter who publishes it. Science needs to be replicated.

The psychology replication crisis is a prime example of why peer reviews and publishing in a journal matters 0.

  • bee_rider  3 days ago

    > The psychology replication crisis is a prime example of why peer reviews and publishing in a journal matters 0.

    Specifically, it works as an example of a specific case where peer review doesn’t help as much. Peer review checks your arguments, not your data collection process (which the reviewer can’t audit for obvious reasons). It works fine in other scenarios.

    Peer review is unrelated to replication problems, except to the extent to which confused people expect peer review to fix totally unrelated replication problems.

  • raincole  3 days ago

    Peer reviews are very important to filter out obviously low effort stuff.

    ...Or should I say "were" very important? With the help of today's GenAI every low effort stuff can look high effort without much extra effort.

bee_rider  3 days ago

Companies produce whitepapers all the time, right? They are typically some combination of technical report, policy suggestion, and advertisement for the organization.

fabianhjr  3 days ago

Most of the world provides funding for research, the US used to provide funding but now that has been mostly gutted.