Comment by carlsverre

15 days ago

Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.

8 comments

carlsverre

Reply
returningfory2  14 days ago

I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built.

My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.

Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.

  • carlsverre  14 days ago

    I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach:

    https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521

    Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

    • quadrifoliate  14 days ago

      I do feel like you're being ganged up on as well. IMO this is a very cool thing that Antithesis is able to find and reproduce this bug in this specific subsystem with a relatively generic workload.

      I guess what would have been an even more cool thing is "we ran some more general testing with Antithesis, and it found five other bugs". Have you thought along those lines or explored something like that? There have to be other, similar bugs lurking in SQLite :)

    • karagenit  14 days ago

      I’m sorry you’re feeling ganged up on, I thought it was a nice article and cool you were able to repro the bug with your tools.

      Like some others have mentioned, one of my earliest thoughts was “how much of a hint was the LLM given about the bug?” I think if the prompt used was stated clearly/verbatim near the beginning of the article, that would probably dispel a good amount of the criticism.

grebc  15 days ago

I read your ad.

It states you replicated the bug once the SQLite team fixed it, and published it.

Not sure what’s difficult about replicating behaviour when it’s spelled out for you.

Mawr  14 days ago

Okay, so use the same methodology to find a new, unknown bug? That'd be actually impressive.

As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.