Comment by docjay

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.

The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.

OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?

“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.