Comment by jwr
2 days ago
I find the word "engineering" used in this context extremely annoying. There is no "engineering" here. Engineering is about applying knowledge, laws of physics, and rules learned over many years to predictably design and build things. This is throwing stuff at the wall to see if it sticks.
Words often have multiple meanings. The “engineering” in “prompt engineering“ is like in “social engineering”. It’s a secondary, related but distinct meaning.
For example, Google defines the second meaning of "engineering" as:
2. the action of working _artfully_ to bring something about. "if not for his shrewd engineering, the election would have been lost"
(https://www.google.com/search?q=define%3AEngineering)
Merriam-Webster has:
3 : calculated manipulation or direction (as of behavior), giving the example of “social engineering”
(https://www.merriam-webster.com/dictionary/engineering)
Random House has:
3. skillful or artful contrivance; maneuvering
(https://www.collinsdictionary.com/dictionary/english/enginee...)
Webster's has:
The act of maneuvering or managing.
(https://www.yourdictionary.com/engineering)
Look up “engineering” in almost any dictionary, and it will list something along those lines as one of the meanings of the word. It is a well-established, nontechnical meaning of “engineering”.
While that may be true, I have a hard time believing that's relevant to the intent of people putting "engineer" into every job title out there.
The same is true of the word 'designer', it's shorthand.
Is this reply not also reply engineering?
Just scanning through I swear I saw the word mathturbation
this
the "engineering means working with engines" gibberish at the bottom is simply dishonest at best
"engineering" means "it's not guessing game"
Thinking through to French and Latin, Engineering means “doing something that is ingenious, and took ingenuity”.
Your posted definitions contradict your conclusion - I would argue there is nothing calculated (as parent poster said, there is no calculation, it just trying and watching what works), artful or skillful (because it's so random, what skill is there to develop?) about "prompt engineering".
I am a cereal eating engineer, while I review the cereal box specification.
I do that every morning, before applying my bus-taking engineering to my job.
Because I do prompt engineering for a living.
So many words lost their meaning today... I am glad I'm not the only one annoyed by this.
Of course, we're all comment engineers here!
If you are going to play that game, "engineering" used to mean that you worked with engines.
Words evolve over time because existing words get adapted in ways to help people understand new concepts.
And in fact, the first engines were developed without a robust understanding of the physics behind them. So, the original version of 'engineering' is more closely to the current practices surrounding AI than the modern reinterpretation the root comment demands.
idk did we call that guessing explode-or-not game "engineering" back then?
maybe we started calling it "engineering" after we had some rules & calculations to make it "not-guessing-game"?
1 reply →
I’ll be damned if Engineering isn’t connected to Ingenuity
https://www.iwc.com/us-en/watches/ingenieur
I still like the Canadian approach that to have a title with the word Engineer in it you have to be licensed by the engineering regulator for the province you work in. The US way of every software dev, mechanic, hvac installer or plumber is an engineer is ridiculous.
Disagree. I think it's valid to describe your work as engineering if it is in fact engineering, regardless of credential. If the distinction is important, call it "<credential name> Engineer". But to simply seize the word and say you can't use it until you have this credential is authoritarian, unnecessary, rent seeking corruption.
Doctors and Lawyers are like this. Maybe something like CPA where you can be an accountant or a certified accountant which you need for something important.
3 replies →
> authoritarian, unnecessary, rent seeking corruption.
Or maybe it's a public service, which reduces instances of fraudulent behavior, and provides cleaner signal in the market of ideas.
1 reply →
How interesting, most people would hesitate to get on a bridge built by a non credentialed civil engineer or have a heart operation by a non certified doctor or get on a 777 with a non credentialed pilot. It’s almost the title the credential the certification signals something about capability
Sorry but in Canada using the word Engineer near your name also means you take legal responsibility personnaly for your professional acts. We are assermented when we earn the title of Junior Engineer after 4 years of university. Then after a period of a few years in the workplace you can have a sponsor Engineer vouch for you. You pass yet another exam and only then you become an Engineer.
This is not true for most so-called Engineers in the US. Anyone can declare themselves an engineer with no exam, no sponsor, no assermentation and no real legal ties to their shoddy work.
3 replies →
> I still like the Canadian approach that to have a title with the word Engineer in it you have to be licensed by the engineering regulator for the province you work in.
That's just not true.
(Despite what Engineers Canada and related parasites tell you.)
hey now don't disparage plumbers they are usually certified and licensed, unlike engineers :P
You could make this same argument about a lot of work that fall onto "engineering" teams.
There's an implicit assumption that anything an engineer does is engineering (and a deeper assumption that software as a whole is worthy of being called software engineering in the first place)
Perhaps. My point is that the word "engineering" describes a specific approach, based on rigor and repeatability.
If the results of your work depend on a random generator seed, it's not engineering. If you don't have established practices, it's not engineering (hence "software engineering" was always a dubious term).
Throwing new prompts at a machine with built-in randomness to see if one sticks is DEFINITELY not engineering.
i dont see where a random seed would have any bearing on "a specific approach, based on rigour and repeatability"
the approach uses random seeds, and the rigours make it repeatable.
if im thinking about mechanical engineering, something like the strength of a particular beam or the cycle life of a bearing is a random number. An engineer's job includes making random things predictable, by apply design tools like safety factors, and observability tools. thats why we prefer ductile materials; over brittle ones. both have a random strength around the spec, but one visibly changes before it fails, where the other doesnt. we can design in inspection processes that accounts for the randomness.
all kinds of tuning operations also start with somewhat random numbers and bring them to a spot. for the very contemporary example: training an ML model. start with random weights, and predictably change them until you get an effective function.
i dont think the randomness excludes "prompt engineering" from being engineering. instead, it's the rigour of the process in turning the random inputs into predictable outputs
It can perfectly be engineering if you have the right validation process. It is, if you can prove that the given randomness can provide satisfactory results to solve the given problem on 99,995% of the cases, then you have a product that solves a given problem following a typical engineering approach.
> Throwing new prompts at a machine with built-in randomness to see if one sticks is DEFINITELY not engineering.
Where does all the knowledge, laws of physics, and rules learned over many years to predictably design and build things come from, if not by throwing things at the wall and looking at what sticks and what does not, and then building a model based on the differences between what stuck and what did not, and then deriving a theory of stickiness and building up a set of rules on how things work?
"Remember kids, the only difference between screwing around and science is writing it down." -Adam Savage
2 replies →
I've seen some good arguments recently that software engineering is weird in that computers ARE completely predictable - which isn't the case for other engineering fields, where there are far more unpredictable forces at play and the goal is to engineer in tolerances to account for that.
So maybe "prompt engineering" is closer to real engineering than "software engineering" is!
With distributed systems I'd say network unreliability introduces a good amount of unpredictability. Whether that's comparable to what traditional engineering disciplines see, I couldn't say. Some types of embedded programming, especially those deployed out in the field, might also need to account for non-favorable conditions. But the predictability argument is interesting nonetheless.
The computer may be reliable but the data passing through it isn’t.
I call it "Vibe Prompting".
Even minor changes to models can render previous prompts useless or invalidate assumptions for new prompts.
Even minor changes to a chemical formulation can render previous process design useless or invalidate assumptions for a new formulation.
Changing the production or operating process in the face of changing inputs or desired outputs is the bread and butter of countless engineers.
I dont think that is a good argument. In chemical engineering world, the provider who just randomly changes formulations would be called "ureliable, shoody, crappy giving us something we not ordered".
Engineers work with non-deterministic systems all the time. Getting them to work predictably within a known tolerance window and/or with a quantified and acceptable failure rate is absolutely engineering.
How do you quantify or decide an acceptable failure rate for llm output?
Same way as any other production model in ML. Or any field that requires quality control. Really, this is not fundamentally different in conceptual approach than implementing any other technology or area of knowledge which is a near verbatim definition of engineering.
Depends on the failure mode and application. But a first approximation is the same way you would for a human output. E.g. process engineering for a support chatbot has many of the same principles as process engineering for a human staffed call center.
> There is no "engineering" here
Prompt engineering is honestly closer to the work of an o.g. engineer. Monitor the dials. Tweak the inputs. Keep the train on time.
I found Hillel Wayne's series of articles about the relationship between software and other engineering disciplines from a few years fairly insightful [1]. It's not _exactly_ the same topic but a lot of overlap in defining wht is "real engineering".
[1] https://hillelwayne.com/post/are-we-really-engineers/
It’s not engineering if you throw anything together without much understanding of the why of things.
But if you understand the model architecture, training process, inference process, computational linguistics, applied linguistics in the areas of semantics, syntax, and more— and apply that knowledge to prompt creation… this application of knowledge from systemic fields of inquiry is the definition of engineering.
Black box spaghetti-hits-wall prompt creation? Sure, not so much.
Part of the problem is the “physics” of prompting changes with the models. At the prompt level, is it Even Possible to engineer when the laws of the universe aren’t even stable.
Engineering of the model architecture, sure. You can mathematically model it.
Prompts? Perhaps never possible.
It changes with any different flavor of a technology in any field of engineering, at least at the level of abstraction that you’re choosing to engage with the problem. Otherwise, this is just machine learning. It yields to the same conceptual approaches in quality control that require fundamental understanding of the underlying fields of study as any area of implementing technology—pretty much the definition of engineering.
You can no more assume the same exact production flow will produce equivalently for a different LLM model than you could for control of a different molecular compound put into product. If you choose only to consider it at the level of equipment assembly then sure, the basic rules of how you assemble the materials— the “physics”— doesn’t hold. If you do so at the same time that such efforts are informed by knowledge of the relevant fields such as material science and of course chemistry then you’re doing chemical engineering. Maybe you don’t want to call the construction workers engineers— though heck in that field many are! But certainly folks like the ones creating the guide posted are being informed by the exact sort of knowledge in the relevant underlying fields.
I agree with you about what's described here.
There is engineering when this is done seriously, though.
Build a test set and design metrics for it. Do rigorous measurement on any change of the system, including the model, inference parameters, context, prompt text, etc. Use real statistical tests and adjust for multiple comparisons as appropriate. Have monitoring that your assumptions during initial prompt design continue to be valid in the future, and alert on unexpected changes.
I'm surprised to see none of that advice in the article.
This article talks about prompt evals https://www.anthropic.com/engineering/writing-tools-for-agen.... There are plenty of approaches to provide some degree of rigor around the slot machine output.
The 'potatolicious rebuttal:
https://news.ycombinator.com/item?id=44978319
(They're not an LLM fan; also: I directionally agree about "prompt" engineering, but the argument proves too much if it disqualifies "context" engineering, which is absolutely a normal CS development problem).
Indeed. Engineering is the act of employing our best predictive theorems to manifest machines that work in reality. Here we see people doing the opposite, describing theorems (and perhaps superstitions) that are hoped to be predictive, on the basis of observing reality. However insofar as these theorems remain poor in their predictive power, their application can scarcely be called engineering.
Is this an AI generated post?
Yes, it was written by a SoTA AGI trained for more than 30 years.
I would like to add that predictable generation defeats the very purpose of generative AI, so prompt engineering in this context will never equate to what engineering means in general.
> Engineering is about applying knowledge, laws of physics, and rules learned over many years to predictably design and build things. This is throwing stuff at the wall to see if it sticks.
There’s one other type of “engineering” that this reminds me of…
> There is no "engineering" here.
Maybe Blahgineering?
"The words "blah blah blah" increase AI accuracy" - https://news.ycombinator.com/item?id=45524413
1) Software engineers don't often have deep physical knowledge of computer systems, and their work is far more involved with philosophy and to a certain extent mathematics than it is with empirical science.
2) I can tell you're not current with advances in AI. To be brief, just like with computer science more broadly, we have developed an entire terminology, reference framework and documentation for working with prompts. This is an entire field that you cannot learn in any school, and increasingly they won't hire anyone without experience.
I saw a talk by somebody from a big national lab recently, and she was announced as the "facilities manager". I wondered for about 5 seconds why the janitor was giving a talk at a technical conference, but it turns out facility meant the equivalent of a whole lab/instrument. She was the top boss.
Unless you are going into a legal definition, where there's a global enumeration of the tasks it does, "engineering" means building stuff. Mostly stuff that is not "art", but sometimes even it.
Building a prompt is "prompt engineering". You could also call it "prompt crafting", or "prompt casting", but any of those would do.
Also, engineering also had a strong connotation of messing with stuff you don't understand until it works reliably. Your idea of it is very new, and doesn't even apply to all areas that are officially named that way.
I hear you. But what's integration in calculus? :)
This tutorial itself is very old. (In recent AI innovation timelines)
Now it's all about Context Engineering which is very much engineering.
First they came for science: Physics, Chemistry, Biology -vs- social science, political science, nutrition science, education science, management science...
Now they come for engineering: software engineering, prompt engineering...
:P
Assume for the sake of argument, that this is literally sorcery -- ie communing with spirits through prayer.
_Even in that case_, if you can design prayers that get relatively predictable results from gods and incorporate that into automated systems, that is still engineering. Trying to tame chaotic and unpredictable systems is a big part of what engineering is. Even designing systems where _humans_ do all the work -- just as messy a task as dealing with LLMs, if not more -- is a kind of engineering.
> rules learned over many years
How do you think they learned those rules? People were doing engineering for centuries before science even existed as a discipline. They built steam engines first and _then_ discovered the laws of thermodynamics.
Same when it's applied to programming though. "Software engineer" has always been a bit silly.
100% I’m old enough to remember when they were called “developers”. Now someone who codes in html and ccs is a “front end engineer”. It’s silly
“Social Engineering”
We live in the dumbest fucking timeline.