Comment by latent-person
25 days ago
In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
And for a similar dplyr-like API on duckdb I recommand the excellent https://ibis-project.org/ !
If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse).
I agree that the API has warts, though typically it is more concise than polars.
The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas.
[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.
3 replies →
I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...
5 replies →