Taken out of context, your post looks like a conservationist who got fed up with pandas being a flagship species and made it their lifelong mission to replace them with polar bears.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
Pandas is more ergonomic in that some ideas can be more tersely represented. The downside is that this results in more dynamism which can change if the underlying data gets updated. Polars is more strict in that it will not silently flip a data type on you. However this strictness does come at the cost of being a bit slower to type and some data idioms not having a good Polars equivalent.
People like to note the speed improvements, but that is the least interesting thing about the library. Rarely have I ever had a problem where I was bottlenecked by Pandas throughout.
Polars is very much a Pandas 2.0 with a bunch of lessons learned. I do not think it is earth shattering changes, but it is worth migrating when you can.
To me, I immediately wonder whether w, x, y, and z here are variables or column names. It would indeed be nice if python could more tersely represent the distinction between a name and a literal string (or worse, as in your R example, a variable reference), but alas. But I think trading some verbosity for explicitness about this distinction is a pretty good trade, and very in keeping with python style.
Polars is a world away from pandas, but I feel that dplyr still offers the most simple and understandable introduction to data analysis for the beginner. The above is a good example of this.
What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy
I agree sql is more elegant. The problems arise when you have to add logic on top of sql. Often I end up constructing queries via string manipulation and that is not very ergonomic. Polars api is more verbose and complex than sql but at least it's not meta-programming.
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
I gave up pandas in favor of polars after someone at work did the same and I am very happy with it. Pandas API is just so much worse and much slower.
Have Polars' inconsistent versioning policy caused you any problems?
What's inconsistent about their versioning?
Taken out of context, your post looks like a conservationist who got fed up with pandas being a flagship species and made it their lifelong mission to replace them with polar bears.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
Thank you, your the only person to correctly interpret my comment :)
You should learn Boa constrictor instead of Python.
I guess I am a casual pandas user only. Reading a guide on migrating/differences, it's hard to see why polars would be obviously better.
Here are a couple reasons:
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
End of the day, they both get the job done.
Pandas is more ergonomic in that some ideas can be more tersely represented. The downside is that this results in more dynamism which can change if the underlying data gets updated. Polars is more strict in that it will not silently flip a data type on you. However this strictness does come at the cost of being a bit slower to type and some data idioms not having a good Polars equivalent.
People like to note the speed improvements, but that is the least interesting thing about the library. Rarely have I ever had a problem where I was bottlenecked by Pandas throughout.
Polars is very much a Pandas 2.0 with a bunch of lessons learned. I do not think it is earth shattering changes, but it is worth migrating when you can.
Both have terrible syntax that make SQL look like the most readable thing ever.
Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data.
Pandas is a mess though.
There's no way SQL is more unreadable than polars. IMO it's the other way around.
2 replies →
Coming from an R/dplyr background, I agree. Compare
df.select(
)
with
df |> select(x, y = w/z)
3 replies →
R really is/was the superior traditional data science language. Python ecosystem is slowly catching up though.
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
1 reply →
Fair point, but you can do something like
`df.select("x", y=pl.col.w/pl.col.z)`
To me, I immediately wonder whether w, x, y, and z here are variables or column names. It would indeed be nice if python could more tersely represent the distinction between a name and a literal string (or worse, as in your R example, a variable reference), but alas. But I think trading some verbosity for explicitness about this distinction is a pretty good trade, and very in keeping with python style.
Polars is a world away from pandas, but I feel that dplyr still offers the most simple and understandable introduction to data analysis for the beginner. The above is a good example of this.
What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy
The verbosity takes a bit to get used too, but it sure beats the anything-goes feeling - messy as you put it - of pandas.
I agree sql is more elegant. The problems arise when you have to add logic on top of sql. Often I end up constructing queries via string manipulation and that is not very ergonomic. Polars api is more verbose and complex than sql but at least it's not meta-programming.
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
You should be using dbt instead of string manipulation for serious query building.
11 replies →
You can query polars data frames with SQL: https://docs.pola.rs/api/python/stable/reference/expressions...
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
7 replies →