Comment by birdsongs
6 hours ago
I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails.
Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.
I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output.
I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser.
I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me.
I spent the time it was churning practicing drawing.
I stopped using GitHub Copilot extension in VS Code when they introduced their new pricing model, but I should have switched much earlier. The developers have barely a clue how LLMs work and the company structure is misaligned with creating a quality product. They do not perform benchmarks to evaluate whether new "features" are any good and instead bloat the context with more and more tools that are rarely useful and often confuse models.
Eventually, the context window got so bloated that they resorted to hiding function bodies in large files, which is of course a stupid idea because then the LLMs have to use other tools to read the files, wasting even more tokens, or hallucinate the content. Honestly, it is amazing that LLMs work at all in VS Code.
You can inspect the context by pressing F1 and then selecting "Developer: Show Chat Debug View" in VS Code (https://github.com/microsoft/vscode/wiki/Copilot-Issues) and marvel at all the garbage that is in there.
Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others context as it tries different solutions. Bonus points for having part 2 fan out on its own, though thats much harder on shared hardware.
Just let the model cook. All of these weird tricks hurt frontier models
> Sure it works fine because you havent yet experienced how much better it can be.
Better in what way, though? It solved my problem, without issue, in a very elegant way. Perhaps I could tweak and optimise to make it solve it slightly faster but honestly, I'm just not that into it. It works, idk, doesn't seem like it needs fixing to me.
But honestly, I don't care a lot about these tools, I said elsewhere I'm probably switching careers if this is the new norm, I hate this version of engineering. It's not even engineering, it's just management.
Only going all in because our company is forcing us to.
Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.