← Back to context

Comment by chis

3 days ago

Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.

2 comments

chis

Reply
boxed  3 days ago

> Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.

If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!