← Back to context

Comment by slopinthebag

11 days ago

yes, your prompts are underspecified relative to the size of changes you are making. you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff, where it makes all sorts of decisions and architectural/functional choices for you (that you didn't even realise needed to be made). if you use a smaller model, it's just not going to be sufficient. you need to work more iteratively with the model, with more steering and more human judgement. astra/opus can do this relatively well in some domains, especially the relatively simple ones like web, agent harnesses, etc. but someone with intention and skill can make use of the less powerful models, it's just a different workflow (and one that I prefer, when I'm not being lazy).

> you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff

Wrong. What I want Luna to be able to do is ~deterministically make small edits or generations of 1~10 LoC according to a specification, and it literally cannot even do that. It violates specifications no matter how small you aim.

  • that doesn't even make any sense. deterministically generating 10loc based on a spec? you should be generating that code based on specific instructions of what you want those 10 lines of code to do, not based on some spec. and there is nothing deterministic about using llm's for codegen. i feel like you're just lost here mate.

    • The specification constrains the implementation details. Repeating the demonstrated example, again: filename must be lowercase is a perfect example of such a specification. You can have a hundred different tasks that require different things but all require that common specification.

      > there is nothing deterministic about using llm's for codegen

      And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.

      5 replies →