← Back to context Comment by esafak 20 days ago What do you suggest they use for ground truth? 2 comments esafak Reply mohsen1 19 days ago I thought about this quite a bit. There are some nuggets in the open source code:- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.- PRs that delete a lot of code. A good change is often deleting code and making things simpler esafak 19 days ago For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.
mohsen1 19 days ago I thought about this quite a bit. There are some nuggets in the open source code:- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.- PRs that delete a lot of code. A good change is often deleting code and making things simpler esafak 19 days ago For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.
esafak 19 days ago For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.
I thought about this quite a bit. There are some nuggets in the open source code:
- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes
- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.
- PRs that delete a lot of code. A good change is often deleting code and making things simpler
For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.