Comment by mlinhares
4 days ago
The downside for formulaic code kinda makes the whole thing useless from my perspective, I can't imagining a case where that works.
Maybe a good case, that i've used a lot, is using "spreadsheet inputs" and teaching the LLM to produce test cases/code based on the spreadsheet data (that I received from elsewhere). The data doesn't change and the tests won't change either so the LLM definitely helps, but this isn't code i'll ever touch again.
> Maybe a good case, that i've used a lot, is using "spreadsheet inputs" and teaching the LLM to produce test cases/code based on the spreadsheet data (that I received from elsewhere)
This seems weird to me instead of just including the spreadsheet as a test fixture.
The spreadsheet in this case is human made and full of "human-like things" like weird formatting and other fluffiness that makes it hard to use directly. It is also not standardized, so every time we get it it is slightly different.
There is a lot of formulaic code that llms get right 90% of the time that are impossible to build macros for. One example that I've had to deal with is language bridge code for an embedded scripting language. Every function I want available in the scripting environment requires what is essentially a boiler plate function to be written and I had to write a lot of them.
There's also fuzzy datatype mapping in general, where they're like 90%+ identical but the remaining fields need minor special handling.
Building a generator capable of handling all variations you might need is extremely hard[1], and it still won't be good enough. An LLM will both get it almost perfect almost every time, and likely reuses your existing utility funcs. It can save you from typing out hundreds of lines, and it's pretty easy to verify and fix the things it got wrong. It's the exact sort of slightly-custom-pattern-detecting-and-following that they're good at.
1: Probably impossible, for practical purposes. It almost certainly makes an API larger than the Moon, which you won't be able to fully know or quickly figure out what you need to use due to the sheer size.
Well yeah, this is a good application of LLMs as this is a fundamentally text-to-text operation they excel at.
But then why are so many people expect them to do well in actual reasoning tasks?
I get that reference! Having done this with Lua and C++, it’s easy to do, but just tedious repetition. Something that Swig could handle, but it adds so much extra code, plumbing and overall surface area for what amounts to just a few lines of glue code per function that it feels like overkill. I can definitely see the use for a bespoke code generator for something like that.
To be pedantic, OP wasn't referencing anything in the usual sense that we use it in (movie, comic, games references). They were more speaking from personal experience. In that sense, there's nothing to "reference" as such.
You could definitely build a code generator that outputs this but definitely a good use case for an LLM.