Comment by augment_me
2 days ago
One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.
The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.
The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.
Overfitting to the input is part of the meta in this type of challenges.
The goal is not to create good, general or maintainable code. The only goal is to produce the fastest code.
Nonetheless gives appropriate context to the speed up for folks like myself who were just skimming.
fair argument
There's only three kinds of optimization realizations
1) Realize that your problem isn't as general as it was
2) Realize that your problem has a better memory model
3) Realize that your problem can be parallelized further
Hyperoptimizing usually falsely leans on 1
This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.
Many open-source communities are wrestling with similar questions. In many ways the generic frameworks end up being reference implementations for the specific solutions to copy and hill-climb towards, and it's as you say unclear what the point is to maintain generic things beyond that.
What I feel is lacking with the solutions that have for example agent-generated and tuned GPU kernels is that the use-cases for them are unclear. If you are a researcher on second-order optimizers, you probably want to be able to handle variable input shapes to experiment, you also might want something readable to understand intermediate steps and perhaps build on that. If you are a neolab running massive training runs for 80% of your VC funding, you need to know that every line in your training code is bitwise identical to the theory/reference because a divergent run from some LLM-generated numerical bug will set you back or bankrupt you, so you can't just plop in a random kernel even if it promises good performance.
So who is the agentically-looped end result for? Except for Openai and Anthropic of course who sold the tools.
You already answered your own question. Testing and trust have always been the most important part.
This has nothing to do with LLMs. There have always been plenty of solutions that are more capable, but untrusted.
Your real question seems to be whether you can prioritize better. What are your project goals? If you have no say in or insight into those goals, you have an even bigger problem. What are you even working on?
For sure, you don't want to be maintaining your dependencies. LLMs make it trivial to rack up insane amounts of technical debt. Why is that appealing to anyone? How is it meaningfully different from the idiots wanting to fork everything on github and copypasta their way to startup success over a decade ago?
Test coverage is important. You need to test a variety of inputs.
Also "will always" is way overconfident. A year ago nothing close to this existed at all. Next year it will fill yet a different role.
this is true. in one of the later problems (cholesky decomposition), the organizer ran the submissions on a tiny training run to validate... and also provided code for same for our reference. most of the top solutions hit 4/8 or so. not very numerically stable.
i found out that as i learnt more domain wise, i was (obviously) able to steer better. doing a re-write can also remove lots of slop and context rot (and subsequently make it easier for both human and LLM to make solution more numerically stable, less reward hackish)
Isn't generalization of solutions what JEPA is trying to solve?