Comment by thefourthchime

4 hours ago

Pac-Man Bench:

Considering the price, no model comes close to being as good as this. However, it did take an extremely long time.

TIME 19m COST $0.16 https://jonclegg.github.io/pacman-bakeoff/#claude-haiku-5-5

All results: https://jonclegg.github.io/pacman-bakeoff/

6 comments

thefourthchime

Reply
myzie  4 hours ago

Interesting that you have gpt-6-luna at $0.01 vs. claude-haiku-5-5 at $0.16 for this task. I see the score disparity though and I played them briefly. My takeaway from this is that the choice between Luna and Haiku 5.5 may remain nuanced. Luna may be a lot cheaper still and good enough for some jobs. Is that your read of the results?

  • thefourthchime  2 hours ago

    Actually, I misspoke. At least as far as Pac-Man Bench, Luna does about as good of a job. The ghost logic's not quite as good, but it also makes a map that doesn't have nonsensical sections in it. So maybe call it a wash.

    • myzie  2 hours ago

      Yeah, I was mainly thinking about how much cheaper Luna appeared to be in this case.

rpcope1  3 hours ago

Something is not right there. DSv4.1 flash shows $1.89 for tens of thousands of tokens? What am I missing?