Comment by spongebobstoes

2 days ago

thanks for your explanation. I'd love to see a concise explanation of the difference in benchmark performance between -preview and -beta

that would help clear the air, if the full-sql version has similar performance to the partial-sql version

3 comments

spongebobstoes

Reply
malisper  2 days ago

The new general version is actually faster than the version accused of cheating.

The architecture of the query engine is:

  SQL -> IR -> Assembly

-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.

  • infogulch  2 days ago

    If it delivers on being faster than the version accused of cheating with reasonable cache settings and no SQL feature limitations, I would say that resolves the accusations, regardless of the source code of -preview.

    The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.