Comment by gtirloni
5 hours ago
> Anyway the point is these models are just about capable of doing the entire job of analyzing a small business and building out all the agents and iterating on them with the business owner.
If you search for LLM benchmarks focused on real-world tasks, you'll quickly learn this ain't the case. No point in telling business owners about that though, they need to see/learn by themselves.
> they need to see/learn by themselves
The cleanup pay is mighty generous as well.
I just finished two solid months of my life cleaning up a vibe coded app.
Has anyone gotten a position as a post-vibe janitor yet?
You got problems there too, but I don't do shitters. That's Engineering.
he is the real world. he is telling you that it is the case. when will you people wake up. this shit is real and happening. people who would have reached out to you asking for help are not now doing that. people from whom you would have asked for help are not getting questions from you anymore.
If only LLM benchmarks could benchmark it in the first day!
Still no Artificial Analysis benchmark yet. Or benchmark for Laguna S 2.1 or Meituan models or lots of other models.
According to a few tasks from my little personal coding benchmark it's very good at coding and kinda bad at web design. (Also excellent at "draw me a picture" one-shot prompts, for whatever that's worth)
On a sneaky one that involved parsing MIME headers and dealing with character encodings it did better than Kimi K3 at Max and for 38% lower cost.
Interestingly it seems noticeably better than the qwen3.8-max-preview model they offered just a few weeks ago.
What does the benchmark even mean when people are using AI to make real world things that solve real world problems?
I see people, and my self making amazing things with AI and fixing old projects and having real world impact at the fraction of the cost it would take me to hire people, or hours spent on my own coding.
I have built tools and systems with AI that have allowed me to build windows drivers, android apps, web apps, iOS apps, vm occultation, custom block drivers, custom file systems and more. To the point where entire products have been created.
Not trying to be a doomsday, but yes. It seems as though with the right infrastructure we are at the point where businesses owners can go from idea to product very fast and not need or hire much external talent.
What amazing windows drivers have you sold?
Idk about sold. But it’s loaded on all the windows machines in a fairly big company that solves a real world problem.
It allows us to apply custom ACLs to AI agents and the child process spawned by AI agents. Giving us the ability to control what files an AI agent can read or write to, while still being in the calling users context. It allows us to force all ai derived processes to use a transparent MITM proxy so we can then also apply robust access rules to remote host allow or deny access to specific urls and not others. It also allows us to monitor access to windows Credential Manager with rules ti allow specific singed binaries to access some credentials but not others. It give us complete control of what AI agents on windows can see or not see or access.
Windows native sandboxing is lacking. You have some stuff in WSL that completely are broken once you call a windows native app. Or you have app containers which are too restrictive and result in applying expensive file system ACL to all files the app containers would access, which can take hours when dealing with million of files, and would be required to be applied every time you chains your app container (there are some workarounds, for them but they still have a one time cost a long with a fairly flaky maintenance process). You can get the network part done by running commands as a different user but that would result in the same file system ACL nightmare that app containers has.
Result is we get seatbelt level sandboxing in windows native, and can apply dynamic rules like preventing access to .aws folders regardless of the OS level ACLs, using glob rules like */.aws, so we don’t have to be aware of the exact path ahead of time.
It also has registry tree ACLs and, can prevent process and process trees from gaining administrative access, the list of features goes on and on.
2 replies →
A better question is which products/services have you avoided purchasing.