Comment by genxy
6 hours ago
I used Fable 5.1 and Opus 5.5 to build a Python interpreter in Rust. Appears to pass all the 3.12 code I throw at it.
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
And also the domain is very verifiable locally.
I wouldn't be surprised if "re-write Python" is in the training set.
You can also consider that open source version of RustPython existed 6 years ago and probably in all LLMs dataset
This sort of ‘it’s probably in the dataset’ thinking is a bit of a relic from the ‘stochastic parrot’ era of LLM critique.
Reinforcement learning doesn’t work by training an LLM on a series of inputs like ‘rewrite python in befunge’ paired with a prepared model implementation. It doesn’t matter if an open source implementation was in the dataset that trained the language model. All it got from that was an appreciation for syntax and language idiom; when it is generating code it’s doing so according to the approaches and patterns that it’s been reinforced into following in the context of a coding prompt - regurgitating existing codebases is undesirable so it is a behavior it will have been reinforced against.
3 replies →
This is really what people conflagrate so often with these tasks, it is basically making the model do recall what could also be a "ant_ag_ds_90917_rwpy9_final" git clone
CPython is in the training data, but this was a tree walking interpreter, so it wasn't just regurgitating cpython. I capped it as 1000 lines for the first 20% of the project, it is currently 13.3kloc and 575k in a single file.
Rewrite X in Y is an ideal RL task given you have X as an oracle trivially.