← Back to context

Comment by Deep-Blue

11 hours ago

As of today NONE of the known AI codebots can solve correctly ANY of the 50+ programming exercises we use to interview fresh grads or summer interns. NONE! Not even level 1 problems that can be solved in fewer than 20 lines of code with a bit of middle school math.

5 comments

Deep-Blue

Reply
cheevly  9 hours ago

I promise you that I can show you how to reliably solve any of them using any of the latest OpenAI models. Email me if you want proof; josh.d.griffith at gmail

  • utopiah  9 hours ago

    I'd watch that show ideally with few base rules though, e.g.

    - the problems to solve must NOT be part of the training set

    - the person using the tool (e.g. OpenAI, Claude, DevStral, DeepSeek, etc) must NOT be able to solve problems alone

    as I believe otherwise the 1st is "just" search and the 2nd is basically offloading the actual problem solving to the user.

    • ehnto  7 hours ago

      > the person using the tool (e.g. OpenAI, Claude, DevStral, DeepSeek, etc) must NOT be able to solve problems alone

      I think this is a good point, as I find the operators input is often forgotten when considering the AIs output. If it took me an hour and decades of expertise to get the AI to output the right program, did the AI really do it? Could someone without my expertise get the same result?

      If not, then maybe we are wasting our time trying to mash our skills through vector space via a chat interface.

NitpickLawyer  7 hours ago

After 25+ years in this field, having interviewed ~100 people for both my startup and other companies, I'm having a hard time believing this. You're either in an extremely niche field (such as to make your statement irrelevant to 99.9% of the industry), or it's hyperbole, or straight up bs.

Interviewing is an art, and IME "gotcha" types of questions never work. You want to search for real-world capabilities, and like it or not the questions need to match those expectations. If you're hiring summer interns and the SotA models can't solve those questions, then you're doing something wrong. Sorry, but having used these tools for the past three years this is extremely ahrd to believe.

I of course understand if you can't, but sharing even one of those questions would be nice.