Comment by prisenco
2 days ago
Everyone's suggestions feel designed to frustrate me. Instructions on how to cajole and plead that seem more astrology than engineering.
This is the pattern I settled on about a year ago. I use it as a rubber-duck / conversation partner for bigger picture issues. I'll run my code through it as a sanity "pre-check" before a pr review. And I mapped autocomplete to ctrl-; in vim so I only bring it up when I need it.
Otherwise, I write everything myself. AI written code never felt safe. It adds velocity but velocity early on always steals speed from the future. That's been the case for languages, for frameworks, for libraries, it's no different for AI.
In other words, you get better at using AI for programming by recognizing where its strengths lie and going all in on those strengths. Don't twist up in knots trying to get it to do decently what you can already do well yourself.
Same. AI is a good tool to use as a sounding board and conversation partner.
I only access claude and others using my browser - I give it a snippet of my code, tell it what exactly I want to do and what my general goal is, then ask it to give me approaches, and their pros and cons.
Even if someone wants to use AI to code for them, its still better to do the above as a first step imo. A sort of human in the loop system.
> It adds velocity but velocity early on always steals speed from the future. That's been the case for languages, for frameworks, for libraries, it's no different for AI.
Completely agree. I'm seeing this in my circle and workplace. My velocity might be a tad bit slower than the rest of my peers when you compare it per ticket. But my long tern output hasn't changed and interestingly, neither has anyone else's.
As an aside, I like your system of completely removing autocomplete unless you need it - may be something like that would finally get me to enable AI in my IDE.
This is the only approach that seems even remotely reasonable
“Prompt engineering” just seems dumb as hell. It’s literally just an imprecise nondeterministic programming language.
Before a couple years so, we all would have said that was a bad language and moved on.
It's technically deterministic, but it feels nondeterministic in chatbots since tokens are randomly sampled (temp > 0) and input is varied. Using the right prompt makes the model perform better on average, so it's not completely dumb.
I like task vectors and soft prompts because I think they show how prompt engineering is cool and useful.
https://arxiv.org/pdf/2310.15916
https://huggingface.co/docs/peft/conceptual_guides/prompting
> It's technically deterministic, but it feels nondeterministic in chatbots since tokens are randomly sampled
Are you not aware the random sampling makes something non-deterministic?
4 replies →
A few years ago we didn't have an imprecise nondeterministic programming language that would allow your mom to achieve SOTA results on a wide range of NLP tasks by asking nicely, or I'm sure people would have taken it.
I think a lot of prompt engineering is voodoo, but it's not all baseless: a more formal way to look at it is aligning your task with the pre-training and post-training of the model.
The whole "it's a bad language" refrain feels half-baked when most of us use relatively high level languages on non-realtime OSes that obfuscate so much that they might as well be well worded prompts compared to how deterministic the underlying primitives they were built on are... at least until you zoom in too far.
I don't buy your past paragraph at all I am afraid. Coding langues, even high level ones, are built upon foundations of determism and they are concise and precise. A short way to describe very precisely, a bunch of rules and state.
Prompting is none of those things. It is a ball of math we can throw words into, and it approximates meaning and returns an output with randomness built in. That is incredible, truly, but it is not a programming language.
5 replies →
Do you mean, like, scripting languages? Are the underlying primitives C and machine language? "Might as well be well worded prompts" is the overstatement of the century; any given scripting language is far closer to those underlying layers than it is to using natural language with LLMs.
6 replies →
> nondeterministic programming language that would allow your mom to achieve SOTA results
I actually think it's great for giving non-programmers the ability to program to solve basic problems. That's really cool and it's pretty darn good at it.
I would refute that you get SOTA results.
That has never been my personal experience. Given that we don't see a large increase in innovative companies spinning up now that this technology is a few years old, I doubt it's the experience of most users.
> The whole "it's a bad language" refrain feels half-baked when most of us use relatively high level languages on non-realtime OSes that obfuscate so much that they might as well be well worded prompts compared to how deterministic the underlying primitives they were built on are... at least until you zoom in too far.
Obfuscation and abstraction are not the same thing. The other core difference is the precision and the determinism both of which are lacking with LLMs.
AIstrology is a term I can get behind.
I think that really captures what I find most irksome about the fanaticism. It's not prompt engineering, it's a statistical 8-ball being shaken until useful output appears.
Just as with any pseudoscience, it can offer a glimmer of usefulness by framing problems in a different way. Just be cautious of who's offering that enlightenment and how much money you may be paying for it.
> how to cajole and plead
When a new person join the team you need to tell them the local coding standard. I don't see why people expect llm to work out of the box instead. The difference is you have to do it at every exchange as llm are stateless.
But yeah I mostly agree with the rest llm work best at very low lewel method by method where you can watch like an hawk they don't introduce silent failure condutions and super high level as system design as reasoning engine, and they still do not so good job at implementing components whole.
Both human and AI should be able to understand the "way we do things around here" by reading the existing code. I could spend an hour telling someone how to write idiomatic code, and they will forget all of it until they actually do some work and see the codebase.
When Claude reads a significant portion of my codebase into context it should be suggesting idiomatic changes. Even if it doesn't initially read a bunch of the codebase to figure out a solution, it should definitely then be trained to do so explicitly to understand the coding standards. Just like a decent dev would do on their own.
if you'd be right that code is so self evident project wouldn't have a coding.md or a contributing.md
The coding standard is quality code and one should bring it with themselves comming into the company. And if you mean linter & formatting rules then if company is not young then their elders had a fist fight to settle once and for all one standard, zip it into a file and everybody just use it
> I don't see why people expect llm to work out of the box instead
because it's not a person? because you have to do it all the time? because of the way literally all other software works?
I don't think anyone would say that the LLMs will produce better code, but they can do it much faster.
I personally did not hit the wall where the use of LLMs would slow me down in the long run.
It has been smooth sailing most of the time, and getting better with newer models.
For me it comes down to "know what you are being paid for".
I'm not a library maintainer. My code will not be scrutinized by thousands of peers. My customer will be happy with faster completion that does the same thing like the more perfect hand crafted version.
Welcome to the industrial revolution in programming. This is the way of things.