← Back to context

Comment by 0x457

6 days ago

Guided generation is called "Structured Output" by other providers?

Well partially generated content streaming thing is great and I haven't seen it anywhere else.

7 comments

0x457

Reply
ApolloVonZ  6 days ago

Sorry if I didn’t use the correct terms. Didn’t catch up on all the terminology coming from my native language. ;) But yes, I agree, the fact that parts, different parameters, of the model can be completed asynchronous by streaming the output of the model, is quite unique. Apple/swift was late with async/await, but putting it all together, it probably plays well with the ‘never’ (I know ) asynchronous and reactive coding.

astrange  6 days ago

An issue with this is that model quality can get a lot lower when you force it into a structured form, because it's out of distribution for the model.

(I'm pretty sure this is actually what drove Microsoft Sydney insane.)

Reasoning models can do better at this, because they can write out a good freeform output and then do another pass to transform it.

  • 0x457  6 days ago

    I have this toy agent I'm writing, I always laugh that I, human, write a code that generates human-readable markdown, that I feed to llm where I ask it to produce a json, so I can parse (by code I, or it wrote) and output in a consistent human-readable form.

    I'm thinking about let it output freeform and then use another model to use to force that into structured.

    • zos_kia  5 days ago

      I've found this approach brings slightly better result indeed. Let the model "think" in natural language, then translate it's conclusions to Json. (Vibe checked, not benchmarked)