Comment by esafak

6 hours ago

I see no 'score' or 'age' mentioned in your script. What does age signify and how are they calculated?

3 comments

esafak

Reply
kristopolous  5 hours ago

This isn't obvious?

    "\(
        10 \* (.codingIndex // 0) | round / 10
    ) \(
      (
        now - (
        .releaseDate |
          try ( strptime("%Y-%m-%d") | mktime )
          catch (now + 86400)
      ) ) / 86400 | floor

Real question. I see 86400 and I know it's time... That might just be me.

I'm not being an ass, I don't know how to talk to people or when I think I'm being clear but I'm actually being cryptic

  • mrbungie  4 hours ago

    It is kind of noisy because the release recency, which is what your "age" column actually represents, is not important data for the comparison you are trying to make.

    Also what message we should get from that table is not really obvious.

    • kristopolous  4 hours ago

      Okay I think there's a familiarity delta. I constantly run into this

      I know artificial analysis quite well as the gold standard in llm evals.

      But I guess they're still obscure

      I didn't think they were.

      The age is important because new techniques keep being developed and so it is a very rough indicator of the size/cost/efficiency trade-off.

      How old a model is is a major indicator of what you can expect from it.

      I really need to develop a better sense for what people know. That's only one of my problems

      Thanks for engaging with me