Comment by AmazingTurtle
13 hours ago
so i did some apples to bananas comparison.
i let 4 different agents review the same dirty code base (200+ files changed, c++, game engine):
- deepseek v4.1 flash via deepseek API: took 20m to find one p1 issue that was relevant and a p2 issue that was irrelevant
- glm 5.3 flash NVFP4 via TP2 DGX Sparks: took 1h 5m to find a p3 documentation issue
- gpt 5.6 sol: took 5m30s for one p1 and one p2 issue. both relevant
- gpt 6.1 sol: took 4m30s for two p1 and one p2 issue. both relevant.
Worth to mention: none of those findings overlapped.
I let deepseek v4.1 flash and gpt 6.1 sol both plan fixes for the summary of the findings. in this particular scenario: gpt 6.1 sol planned to increase a buffer size from 128 to 256 to fit the 191 elements that blew past a test scenario. it included some headroom. deepseek v4.1 flash on the other hand decided to make it exactly 191 to satisfy the test, for example.
The TPS decode on deepseek is impressive. I never had a second / feeling of being left aone and though "damn this thing is working". Despite all my efforts, it took ~4x longer in a review, found less and planned worse than GPT 6.1 Sol. Wall clock time. I don't have token counts for comparison. If I were to grade those models from 1-10 (10 = top tier GG), I'd give deepseek a solid 6.5 and 6.1 sol a 7.5-8.0 maybe.
LLMs are non-deterministic, you could run the same prompt on the same code base with the same models, and have different result. I don't think only one run with each model is enough to draw conclusions.
We have to keep remembering this for some reason.
New model comes out and someone says, "it can (or can't) draw ducks on bikes" or some esoteric thing. But it's not enough statistical power to claim anything other than a personal anecdote.
I got a flat tire in downtown today. Therefore, driving in downtown causes flat tires.
Honest question about the non-deterministic nature of llms. Every programming language provides some way to control the random seed. Is this not true in GPU programming land?
It doesn't really matter for most purposes. Determinism doesn't make them any less chaotic, it's not like fixing the seed makes the output any more predictable. Some percentage of seeds will behave in ways you don't like for a particular input, and you can't know which beforehand.
Yes, but for one thing, floating-point math is non-associative so even if you use the exact same random seed, you can get different results on different implementation kernels.
For another, the dependence on a random seed is just one example of how you can get different behavior. You could just as well get different results due to e.g. a tiny difference in prompt wording or punctuation or whitespace.
If a model passes some test on 70% of random seeds, and fails on the other 30%, then knowing the rough probability of success is much more useful than knowing one random sample of the outcome.
And empirically this kind of probabilistic success is quite common, because many LLM benchmarks report "pass@k" success rates that are much higher than the "pass@1" success rate on a single attempt.
1 reply →
you probably tried with DS on max and I don't blame you.
But try it on high instead of max if possible. max is known for being overly verbose to the point of being detrimental compared to high at times.
For example Simmons svg pelican took 11 minutes on max and 1 minute on high. And they are comparable.
With DS Flash it doesn't fall apart below xhigh like GPT Luna models do?
In your case, is it a public repository that the llms might have found the issues for online?
I too am thinking about creating some benchmarks that I can understand myself. The LLM 'cheating' by finding known issues through some sort of history or online source is a path you want to protect against. Then again, you DO want them to be able to look online for people having similar issues.
How do you get dirty changes into a public repo online?
> Worth to mention: none of those findings overlapped.
This is why using multiple model families is important. Don't limit yourself by being a stan for this or that company.
Especially when $10-$20 on OpenRouter with e.g. DeepSeek (or any other inference provider, or their official API) will get you very, very far.
Gets me about a day or two lol
That may be the case, but simply running the same code review twice with the same model may give very different results as well. Impossible to know with the information we have.
> Worth to mention: none of those findings overlapped.
Yet to be obsolete.
If none of the findings overlapped then LLMs are either
- For some reason choosing only a subset of issues.
- Different LLMs actually learn different things.
I suspect the former since they do this for me and it's extremely annoying sometimes
3 - non deterministic, lazy
What is p1, p2?
This read like a conversation you were having in your head -- unfiltered.
Very commonly used acronyms in business/SWE contexts.
The lower the number the higher the priority, e.g. if production is broken someone might say "This is p -1" (higher than your highest priority).
P0 is "do this immediately", P1 is very important, P2/P3 are like "nice to haves." P100 would mean something like "we will never do this"
Same understanding as you.
At 3am,
P0 = get people out of bed and fix this now.
P1 = we can fix this in the morning.
P2 = we can fix this before CoB today.
P3 = push it into the current sprint.
P4 = put it in backlog.
priority 1,2 etc, more like severity of the cases
It’s a very common abbreviation; i had no issue understanding GPs post tbh
Was that the old DS 4.1 Flash or 4.1 Flash 1017?
I tried it out today, was the new one: 1017.
What reasoning effort?
Flagship gpt models are crazy efficient. 6.1 Sol is like a savant that looks at a problem and spits out the answer seemingly without thinking.
Opposite of my experience - openai models, and codex are clearly the slowest (out of 5 different models/harnesses I use on a regular basis).
[flagged]