← Back to context

Comment by mmh0000

6 days ago

LLMs are good at producing what they/the public know.

In this case:

  LLMs know the USB Spec very well.

  LLMs know how to read raw packet dumps.

  LLMs know how to convert a packet dump to USB spec

  LLMs know how to write code to generate USB packets from the spec.

LLMs are also VERY good at transliteration, i.e., converting known-good Python to Rust.

Basically, If you have a well-documented problem, the LLM is a shortcut to learning it yourself. LLMs fail when you have a novel or poorly documented problem. They also fail when you provide the LLM with terrible context or too much context.

Don't sell in-context learning short. Right now I'm waiting on Claude to wrap up the latest of a half-dozen extensive changes to XML files for a fairly-obscure (and obsolete) closed-source electronics CAD program. I am pretty sure it doesn't know anything about these files besides what's in the XML .DTD file (which I also gave it.)

This is a very novel, reasonably-poorly-documented problem, and so far it has batted 1.000.

  • Kinda sounds like you are finally realizing the dream of XML and DTD envisioned 25 years ago.

    • Exactly! Not to mention the belated realization of the so-called "expert systems" we were promised back in the 80s. Better late than never.

      1 reply →