Comment by kristopolous
3 days ago
I use local models + openrouter free ones.
My monthly spend on ai models is < $1
I'm not cheap, just ahead of the curve. With the collapse in inference cost, everything will be this eventually
I'll basically do
$ man tool | <how do I do this with the tool>
or even
$ cat source | <find the flags and give me some documentation on how to use this>
Things I used to do intensively I now do lazily.
I've even made a IEITYuan/Yuan-embedding-2.0-en database of my manpages with chroma and then I can just ask my local documentation how I do something conceptually, get the man pages, inject them into local qwen context window using my mansnip llm preprocessor, forward the prompt and then get usable real results.
In practice it's this:
$ what-man "some obscure question about nfs"
...chug chug chug (about 5 seconds)...
<answer with citations back to the doc pages>
Essentially I'm not asking the models to think, just do NLP and process text. They can do that really reliably.
It helps combat a frequent tendency for documentation authors to bury the most common and useful flags deep in the documentation and lead with those that were most challenging or interesting to program instead.
I understand the inclination it's just not all that helpful for me
This is a completely different thing to AI coding models.
If you aren't using coding models you aren't ahead of the curve.
There are free coding models. I use them heavily. They are ok but only partial substitutes for frontier models.
I'm extremely familiar with them.
Some people, with some tasks, get great results
But me, with my tasks, I need to maintain provenance and accountability over the code. I can't just have AI fly by the seat of its pants.
I can get into lots of detail on this. If you have seen tools and setups I have done you'd realize why it doesn't work for me.
I've spent money, the results for me, with my tasks, have not been the right decision.
> I'll basically do
or even $ cat source | <find the flags and give me some documentation on how to use this>
Could you please elaborate on this? Do I get this right that you can set up your your command line so that you can pipe something to a command that sends this something together with a question to an LLM? Or did you just mean that metaphorically? Sorry if this is a stupid question.
Yes, I use simonw's `llm` for that: https://github.com/simonw/llm
Example:
I'm not the OP, but I did build a tool that I use in the same way: https://github.com/scottyeager/Pal
Actually for many cases the LLM already knows enough. For more obscure cases, piping in a --help output is also sometimes enough.
i guess op means: $ man tool | ai <how do I do this with the tool>
where ai could be a simple shell script combining the argument with stdin
Is your RAG manpages thing on github somewhere? I was thinking about doing something like that (it's high on my to-do list but I haven't actually done anything with llms yet.)
I'll get it up soon, probably should. This little snippet will help you though:
That goes man -> html -> markdown which is not only token efficient but also llms are pretty good at creating hierarchies from markdown
I bet you could do the same thing with pandoc and skip serializing to HTML entirely.
1 reply →
Not the OP, but I did release my source :D https://github.com/scottyeager/Pal
My tool can read stdin, send it to an LLM, and do a couple nice things with the reply. Not exactly RAG, but most man pages fit into the context window so it's okay.
I use llm from command line too, time to time, is just easier to do
llm 'output a .gitignore file for typical python project that I can pipe into the actual file ' > .gitignore
> My monthly spend on ai models is < $1
> I'm not cheap
You're cheap. It's okay. We're all developers here. It's a safe space.
While I say this somewhat in jest, frugal is just cheap but with better value.
this is the extent to what I use any LLM - they're really good at looking up just about anything, in natural language, and most of the time even the first hit, without reprompting, is a pretty decent answer. I used to have to sort thru things to get there, so there's definitely an upside to LLMs in this manner.
Have you looked at tldr/tealdeer[0]? It may do much of what you're looking for, albeit without LLM assistance.
0: https://tealdeer-rs.github.io/tealdeer/
> I'm not cheap, just ahead of the curve.
I'm not convinced.
I'm convinced you don't value your time. As Simon said, throw $20-$100/mo and get the best state of the art models with "near 0" setup and move on.