Comment by jbverschoor
2 years ago
It's funny to see that a lot of services are finally moving from a human-readable / plain text format towards structured protocols right at the point where we can finally have LLMS parse the unstructured protocols :-)
Well you can't really trust an LLM to give you reproducible output every time, you can't even trust it to be faithful to the input data, so that's nice to have a standard format now. And for like a millionth of the computing resources to parse it. Also Whois was barely human-readable, with the fields all over the place, missing or different from one registry to the other. A welcome change that should have come really sooner.
we can't ever have LLMs reliably parse any form of data. You know what can parse it perfectly though? A parser. Which works perfectly, and consistently.
Except that the whole problem of WHOIS, that RDAP is solving, is that a WHOIS response an unstructured plaintext response, that's entirely arbitrarily formatted according to the whims of the TLD manager.
Ever tried to parse WHOIS data? You literally have to write a parser per TLD.
And things get even more stupid when you start talking about WHOIS records for IP ranges. Then you have to write a parser per IP-range delegation — starting at IANA, and working recursively, all the way down to the individual ASN. Where you have no idea how many delegating parties are going to be involved — and so get their own step in the chain, formatted however they wish — for any given IP address. (Ask me how I know.)
> Which works perfectly
... on conformant inputs, when it has no bugs.
On non-conformant inputs, a parser will barf and yell at you, which is exactly what you want.
On non-conformant inputs, there's absolutely no telling what an LLM will do, which is precisely the problem. It might barf, or it might blissfully continue, and even if the input was right you couldn't remotely trust it to regurgitate the input verbatim.
As for bugs, it is at least theoretically possible to write a parser with no bugs, whereas an LLM is fundamentally probabilistic.
Of course we can. Reliability is a spectrum, not a binary state. You can push it up however high you like, and stop somewhere between "we don't care about error rate this low" and "error rate is so low it's unlikely to show in practice".
It's not like this is a new concept. There are plenty of algorithms we've been using for decades that are only statistically correct. A perfect example of this is efficient primality testing, which is probabilistic in nature[0], but you can easily make the probability of error as small as "unlikely to happen before heat death of the universe".
--
[0] - https://en.wikipedia.org/wiki/Primality_test#Probabilistic_t...
There are two problems with this comparison. First, probabilistic prime generation has a mathematically proven lower bound that improves with iteration. There is no comparably robust tuning parameter with an LLM. You can use a different model, you can use a bigger variant of the same model, etc., but these all have empirically determined and contextually sensitive reliability levels that are not otherwise tunable. Second, the prime generation function will always give you an integer, and never an apple, or a bicycle, or a phantasm. LLMs regurgitate and hallucinate, which means that a simple error rate is not the only metric that matters. One must also consider how egregiously wrong and even nonsensical the errors can be.
12 replies →
If you job is to be a referent, to have authority. You absolutely don't want to make any error. Pretty safe isn't enough, you need to be absolutely sure that you control the output.
You only have one job, don't delegate authority.
But isn't using LLM for that really expensive? Seems wasteful.
I wouldn't use LLMs, but if I did, I would try to get the LLM to write parser code instead.
If it can convert from one format to another, then it can generate test cases for the parser. Then hopefully it can use those to iterate on parser code until it passes the tests.
In a sense, asking it to automate the work isn't as straightforward as asking it to do the work. But if the approach does pan out, it might be easier overall since it's probably easier to deploy generated code to production (than deploying LLMs).
My desktop GPU can run small models at 185 tokens a second. Larger models with speculative decoding: 50t/s. With a small, finetuned model as the draft model, no, this won't take much power at all to run inference.
Training, sure, but that's buy once cry once.
Whether this means it's a good idea, I don't think so, but the energy usage for parsing isn't why.
A simple text parser would probably be 10,000,000 times as fast. So the statement that this won't take much power at all, is a bit of an overstatement.
50 tokens per second. Compared to a quick and dirty parser written in python or even a regex? That's going to be many many orders of magnitude slower+costlier.
1 reply →
In order to make the point that
> energy usage for parsing isn't why
You'll need to provide actual figures and benchmark these against an actual parser.
I've written parsers for larger-scale server stuff. And while I too don't have these benchmarks available, I'll dare to wager quite a lot that a dedicated parser for almost anything will outperform an LLM magnitudes. I won't be suprised if a parser written in rust uses upwards of 10k times less energy than the most efficient LLM setup today. Hell, even a sed/awk/bash monstrosity probably outperforms such an LLM hundreds of times, energy wise.
6 replies →
It’s not just about the energy usage, but also purchase cost of the GPUs and opportunity cost of not using those GPUs for something more valuable (after you have bought them). Especially if you’re doing this at large scale and not just on a single desktop machine.
Of course you were already saying it’s not a good idea, but I think the above definitely plays a role at scale as well.
2 replies →
My Atom n270 netbook with mawk and a few lines parsing the files with a simple regex will crush down your GPU+LLM's on both time and power usage.
My assumption is that models are getting cheaper, fast. So you can build now with OpenAI/Anthropic/etc and swap it out for a local or hosted model in a year.
This doesn't work for all use cases but data extraction is pretty safe. Treat it like a database query -- a slow but high availability and relatively cheap call.
While it will become cheaper, it will never be as fast / efficient as 'just' parsing the data the old-fashioned way.
It feels like using AI to do computing things instead of writing code is just like when we moved to relatively inefficient web technology for front-ends, where we needed beefier systems to get the same performance as we used to have, or when cloud computing became a thing and efficiency / speed became a factor of credit card limit instead of code efficiency.
Call me a luddite but I think as software developers we should do better, reduce waste, embrace mechanical sympathy, etc. Using AI to generate some code is fine - it's just the next step in code generators that I've been using throughout all my career IMO. But using AI to do tasks that can also be done 1000x more efficiently, like parsing / processing data, is going in the wrong direction.
6 replies →
Still, I wouldn't use an LLM for what's essentially a database query: by their very nature, LLMs will give you the right answer most of the times, but will sometimes return you wrong information. Better stay on a deterministic DB query in this case.
As usual, arguments for LLMs are based on rosy assumptions about future trajectory. How about we talk about data extraction at that point in the future when models are already cheap enough. And in the meantime just assume the future is uncertain, as it obviously is.
[dead]
deepseek API costs are quite literally pennies per million tokens
Which world would you rather live in: * structured protocols that can be parsed by machines * unstructured protocols that are unreliably parsed by LLMs that require significant power and latency
In addition to ~determative machines and LLMs, what about humans reading the data?
https://deviq.com/antipatterns/shiny-toy