Comment by m_ke
2 months ago
What a lot of people don’t know is that SWE-bench is over 50% Django code, so all of the top labs hyper optimize to perform well on it.
2 months ago
What a lot of people don’t know is that SWE-bench is over 50% Django code, so all of the top labs hyper optimize to perform well on it.
I know python is more prevalent in SWE-Bench than any other language, but more than 50% django sounds like a big stretch. Citation?
Edit, it's about 37%, and python-only. https://arxiv.org/pdf/2310.06770v3