Comment by olalonde

2 days ago

Recently, Gemini helped me fix a bug in a PPP driver (Zephyr OS) without prior knowledge of PPP or even driver development really. I would copy-paste logs of raw PPP frames in HEX and it would just decode everything and explain the meaning of each bytes. In about an hour, I knew enough about PPP to fix the bug and submit a patch.

https://g.co/gemini/share/7edf8fa373fe

12 comments

olalonde

Reply
skydhash  2 days ago

Or you could just read the PPP RFC [0].

I’m not saying that your approach is wrong. But most LLM workflows are either brute forcing the solution, or seeking a local minima to be stuck in. It’s like doing thousands of experiments of objects falling to figure out gravity while there’s a physics textbooks nearby.

[0]: https://datatracker.ietf.org/doc/html/rfc1661

  • olalonde  2 days ago

    Ironically, I could’ve read all 50 pages of that RFC and still missed the actual issue. What really helped was RFC 1331[0], specifically the "Async-Control-Character-Map" section.

    That said, I’m building a product - not a PPP driver - so the quicker I can fix the problem and move on, the better.

    [0] https://datatracker.ietf.org/doc/html/rfc1331

  • wrasee  2 days ago

    I could also walk everywhere, but sometimes technology can help.

    There’s no way I could fully read that RFC in an hour. And that’s before you even know what reading to focus your attention on, so you’re just being a worse LLM at that point.

    • Retric  2 days ago

      The difference is you’d remember some of the context from reading the thing where an LLM is starting from scratch every single time it comes up.

      5 replies →

tralarpa  2 days ago

Interesting that it works for you. I tried several times something similar with frames from a 5G network and it mixed fields from 4G and 5G in its answers (or even from non-cellular network protocols because they had similar features as the 5G protocol I was looking at). Occasionally, the explanation was completely invented or based on discussions of planned features for future versions.

I have really learned to mistrust and double check every single line those systems produce. Same for writing code. Everything they produce looks nice and reasonable on the surface but when you dig deaper it falls apart unless it's something very very basic.

  • foobarian  2 days ago

    Similarly I found the results pretty mixed whenever a library or framework with a lot of releases/versions is involved. The LLM tends to mix and match features from across versions.