Comment by gonzalohm
6 days ago
In my opinion, the problem is not even the cost. The problem is that people are using AI for running recurrent stuff instead of writing code to automate it.
For example. Imagine that you are comparing two documents (let's assume diff doesn't exist). You could ask an AI to compare the differences from you or you could use AI to write a tool to do it. For whatever reason, people are starting to go with the former not realizing that now they basically have to pay to compare documents.
I have exposure to AI initiatives at several companies including a few F500's. I have seen teams dump huge logs into frontier models that took hours to get so-so results that we were able to replace with a few lines of python code at 1000 times the speed and 100% accuracy. When asked why they were doing this they literally said "because we don't understand the subject matter so we were depending on the AI". I saw one team file a complaint with a vendor about a frontier backed coding harness and it's inability to consistently format headers because they were using it as a reporting engine. When I recommended they just use the coding tool to write code to generate reports you would have thought I had just cured cancer from their response. I frequently see people complain about the fact that AI is going to take their jobs and then see them gripe about the fact that AI is 'worthless' because it can't do more of their job than it already does. It's easy to see the difference between the people seeing 10x productivity gains from leveraging AI and those who aren't and it's not the AI.
i have trouble understanding these situations, e.g. the AI itself would presumably make the suggestion to write a python script for such a task. It seems to me that there two huge problems right now * understanding which category of problems an LLM is an appropriate solution for (rather than throwing LLMs at any and all problems) * matching model capability (and therefore cost) to the problem at hand. You can easily overspend massively by using a model that's too powerful
Someone asked me if I was using models for fantasy sports, and if it was smart enough to help make decisions about drafting.
My answer: no, but it was able to help me find the website and social handles for every beat writer for every team, and generate a simple website where I can do a daily skim of teams/players and draw my own conclusions.
LLMs are a tool, not a panacea.
I've heard this framed as "AI raises the floor by 2x or less but raises the ceiling by 10x or more"
Laziness, pure and simple. The inevitable consequence of “the LLm is the compiler now”. And what do you even expect people to do when they are forced at threat of termination to use AI for everything as much as possible? Not to mention people are being pressured to do insane thing like review hundreds of pull requests per day and deliver like 15 features per week so OBVIOUSLY there isn’t time to build out proper tooling. Just shove everything in a prompt and call it a day. Some people have families to feed, just do what you’re told.
Agreed. I’ve been telling my team to build up internal packages so we can push all that ad hoc reinvention into something more tangible and deterministic. Invest the $$$ in inference into something the agent can reach for next time that’s neutral and consumable by other code to reduce future spend.
Yes. Build compact CLI-driven tools, write a skill for it (you can use your agent to do most of this work for you).
It just requires being willing to think instead of mashing prompts into a keyboard.
Because you look at the work from the perspective of a programmer, not the perspective of a regular person.
Normal people have never gone around automating their work. The most automation they do is dynamic tables on excel sheets.
I obviously know building a tool that can programmatically do something is a better solution, but I think that requires a fundamental shift in how people work. People need to be told by someone "this is how you should be using the AI" but right now they're simple told "use the AI".
I'm talking about programmers doing this. That's what's sad. These were normal people before,but it feels like they have some kind of AI schizophrenia now. They don't use their brains anymore
Same, even opus favor short term solution and scripts with a billion flags that constabtly require rescanning to understand how to launch it is a constant struggle to get it to build sane default and reusable scripts that run with minimal parameters
Yeah, and what's up with adding dry run to everything? I saw some code that doesn't write anything but still the AI added a dry run which had a completely different codebase
Because dry run is in a lot of scripts in its training data. It's not "thinking" about the script or the concept of a dry run.
And everything configurable gets an environment variable. Editing the first few lines in the script is a fine way to configure things in Python.
AI can do things around semantic analysis that a deterministic diff tool cannot.
I understand and agree with your point though.
I'm curious if you could give me an example of something that couldn't be down deterministically. We have fuzzy search/matching too ? Regex is a monster when used correctly.
A model can 'analyze' the intent of a patch, 'understand' it, and then correctly merge it in a derived codebase, going further than merely resolving conflicts.
I sometimes find myself with thousands of log lines from a problematic execution and a known good reference, wondering nonspecifically if "something weird" happened in the first one. I don't think there's any matching-based solution there; you need a scan process that understands variations in execution time, object identifiers, etc. aren't meaningful.
2 replies →
Pretty much anything for which you'd need intelligence of any kind. Questions such as: Do these two paragraphs have the same semantic meaning? Do they have the same sentiment? Do these two methods have the same contract? etc. Not all documents our code and even with code deterministic tools gets you only so far.
It's this and worse. To use your example, it's like people using AI to write a diff algorithm, incorrectly, then using AI to fix it, because they don't know that diff exists already. Lazyness and starting development with a very low level of understanding. People think lowering the barrier to entry is a good thing, when in reality there are just fundamentals and things you just have to know before you can start using a tool like llms properly.
Isn't that the supposed point of it though? At least how it is marketed/hyped. Don't use your brain, you don't need one, spend all your thinking energy on... dunno, something else, and leave all the "mundane" stuff to AI. Just pay for the tokens, it's going to make you 10x more efficient, the $1000/month is worth it.
100% this. For my own company I mostly build deterministic workflows that may have a simple AI step in the middle using an appropriate Chinese model in a very limited way. I wouldn't want to burn tokens to satisfy some metric.
With this AI is a fallback and not the default. Sounds like large companies have it backwards.
I agree, but even this use case isn't the most wasteful. The interwebs says Agentic consumes 50% of token use, but I'd hazard this number is north of 90% for many shops. My cynical view of Agentic is its sole purpose is to make "number go up".
Look at me! I'm the smartest guy. I've wasted 10M tokens! No one has wasted more!
Same with writing boilerplate code. It’s been a solved problem yet here we are.
Recurrent expensive inefficient processes? Sounds familiar !
This is why we have business analysts and software developers.
To help identify inefficiencies and to build technical solutions.
Oh no! People are doing what they've been told to do!
[dead]
it's all about cost at the end of the day. if you're allowed and encouraged to tokenmaxx, then of course this'll happen.