Comment by blueblazin
8 hours ago
With every new model I've been attempting to build something that takes at least 1 month and really pushes the model. So far I've done:
- Python interpreter in Rust (fail): https://github.com/BlueBlazin/pyrs
- Low-poly Google Earth like engine (success): https://github.com/BlueBlazin/lowpolyearth
- Codebase -> 2d infinite-zoom visualization (success): https://github.com/BlueBlazin/blockviz
I have never understood or liked all the obsession with one-shot builds.
> https://github.com/BlueBlazin/lowpolyearth
Publishing screenshots and the README on GitHub but not the code is sneaky at best, deceiving at worst.
I’m not sure why they don’t just open the source (are they planning on monetizing it?), but GitHub has a convenient issue tracker that everyone already has a user account for.
I’ve built a retro operating system and programming environment for ESP32 microcontrollers called BASIC32.
I took it to 39% complete, as a language, over 8-weeks. Then, I spent just 50 minutes describing all the missing parts and it finished the language in 1-shot. I didn’t think it would work. It did.
I used Opus 5.5 on a $20 plan and I didn’t even hit my limits.
I have a feeling that the 39% served as a good framework for the LLM to work on the remaining part while keeping it consistent with the existing structure and style. So, in a sense, your 8 weeks are a big part of why it succeeded.
Yeah I've experienced that at work. I see a lot of comments on here saying you can't one shot features in real world software, but we do, and yes we still have a human review the code with a high bar.
But we've also spend a lot of effort over the past year and a half to make the codebase and tooling more approachable to LLMs so they know how we like to do things and follow our standards.
I used Fable 5.1 and Opus 5.5 to build a Python interpreter in Rust. Appears to pass all the 3.12 code I throw at it.
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
And also the domain is very verifiable locally.
I wouldn't be surprised if "re-write Python" is in the training set.
You can also consider that open source version of RustPython existed 6 years ago and probably in all LLMs dataset
4 replies →
This is really what people conflagrate so often with these tasks, it is basically making the model do recall what could also be a "ant_ag_ds_90917_rwpy9_final" git clone
CPython is in the training data, but this was a tree walking interpreter, so it wasn't just regurgitating cpython. I capped it as 1000 lines for the first 20% of the project, it is currently 13.3kloc and 575k in a single file.
Rewrite X in Y is an ideal RL task given you have X as an oracle trivially.
I’ve also been exploring a visual way to understand a code base and have explored turning it into a virtual city.
https://app.preview.craftspec.com/app/city/
This whole thing is just a proof of concept, and needs a mouse to interact with.
It’s incredible what AI has made possible with experimental projects that would require months of coding.
This is lovely
That's a really cool concept! I hit a bug where there were two things labeled "processing" and then it jumped to the wrong one when I clicked it.
Oooh, nice catch, I'll feed that back to get fixed.
100% agreed.
There's so many youtube channels getting hundreds of thousands of views on EVERY new major model release just doing essentially zero effort one-shots, and people lap it up as if building things that already exist is somehow the best their imagination allows for.
I do love being able to test a model for 3+ weeks (>1 month is too long personally for my budget) and actually learning how to build scalable software and actually develop your "engineering" skills.
The 'blockviz' is same kind-of project i tried to create as well (c4 level model zoomable etc.) so i was surprised to see same kind-of project popup - but this is I think also an 'issue' with these model-created-software.. when you reach'your' vision its already done: for example I clicked on the github link, expected a clear About Install Faq kind of README.md but all I get is the model-created explaination :) can't just download some app and use it without friction. I see this a lot.
You can click the big image or the url at the right and use it in browser.
https://blueblazin.github.io/blockviz/
I tried it on some of my code and it's pretty neat, though not fully intuitive to follow more complex code structures for me.
yeah, one-shot (medium-large) builds just don't seem to give me good results yet. I use them now for small examples though, e.g. for visualizing a process. Most recently I used it to visualize neural networks, and how convolution works. That might not be something most people need, but it really helped me.
Hah, found someone else also using it for visualizations of algos! I've used it to show me how quad-trees[0] work and I'm impressed with the results. One-shotted too so I agree one-shots are best for this kind of very specific scenario.
[0] https://jsfiddle.net/3yLrsem0/
> I've been attempting to build something that ... pushes the model ... never understood ... one-shot builds
Same, and I have found some of the failure modes in the realm of eyes free coding and limited auditing. I often see it more as pushing the process than the model, as I tend to bounce around models for the benefits diversity provides.
Have you also found there is a point where they hit code complexity and start churning without progress? (at which point you have to step in to help the bot step back and think big picture, which usually takes a few sessions)
Yes! I absolutely have! And I've been thinking a lot lately about how to prevent them from collapsing into this state. It is getting better with newer and better models though. But yes, there comes a point when the codebase size and complexity crosses a threshold and we enter a churn state with very little progress.
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
Have you seen this with Opus 5.5?
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
2 replies →
I just have it constantly update real-world analogues for keeping track of everything. I have it create guides, note cards, flowcharts, running to-do lists, etc. with every change.
It helps us keep track of things, why not AI, too?
> How can we teach "systems thinking" to a model?
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.
[flagged]
[dead]