← Back to context

Comment by walrus01

11 hours ago

I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.

https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...

3 comments

walrus01

Reply
guregodevo  11 hours ago

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

  • wredcoll  6 hours ago

    This is somewhat tangential but speaking of ai benchmarks, I want to see one benchmarkes against passing those "filelocker" websites used for illicit downloads.

    You know, the kind with 50+ ads on the page and 8 of them say download and you have to click it 5 times to get past the ads then do that on two more pages?