Comment by claytonjy
3 years ago
Can someone ELI5 why pypy doesn't or can't work with C-based packages like numpy or psycopg? I know nothing of how pypy does its magic.
If we could use pypy, while still using those packages, I think it'd be the go-to interpreter. Why can't pypy optimize everything else, and leave the C stuff as-is?
How does pypy handle packages written in other languages, like rust? can I use pypy if I depend on Pydantic?
Basically afaik the default C API (`Python.h`) is matched to CPython's internal representations, hence it's a pain to support it for alternative implementations and incurs cost penalties. The preferred way to interact with C code in pypy is through cffi (https://cffi.readthedocs.io/en/latest/) and ctypes (which afaik is implemented in pure python on top of cffi in pypy).
Numpy being itself written in C and C++ it is strongly tied to the C API and has a complicated build process. Some stuff works and some don't (didn't try recently). If you're invested in numerical python you should most likely not use pypy but go for stuff like cython (like scipy does).
For psycopg apparently you can use psycopg2cffi (never tried).
> How does pypy handle packages written in other languages, like rust? can I use pypy if I depend on Pydantic?
PyO3 supports pypy so everything should be fine.
Lots of questions :)
For c-extensions see https://www.pypy.org/posts/2018/09/inside-cpyext-why-emulati...
We would like to be able to "just JIT" better. But for that we need feedback about what is still unreasonably slow, and resources to work on improving it. Right now PyPy is on a shoe-string budget of volunteers.
For rust, like CPython, use PyO3, which works with PyPy.
I am not sure about Pydantic. Sounds like a topic for someone to investigate on their codebase and tell us how PyPy does.