Comment by trombonechamp
3 days ago
Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.
It's standard sql behavior, users always specify the ordering they want as part of the query.
Often people new to databases and SQL thinks ordering is implicit, I have teach dozens of juniors that believe this is the default. I always wondered why that is the thought process...
Because out of the first 100 queries they ran, 99 came back in order and they didn't notice that the 100th didn't.
If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit.
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
Unfortunately (for your case) the ordering of group_by, join, and unique all run in parallel hash aggregation across the threads so the output order comes about by how it gets partitioned across cores. Which is why you can get different order of rows depending on the machine even when you have the same set of data and polars version (this has happened to me). To fix you can set maintain_order=True or probably better an explicit sort whereever you save or compare / diff the output.
This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims. Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
By "implicit ordering", do you mean "implicitly assumed that the data is ordered a certain way"? Since if that assumption of data being sorted a certain way is broken on some systems and not others, the result might be both non-deterministic (which could be a bug if the result is not allowed to be non-deterministic, but may or may not be a bug regarding the algorithm's assumptions) as well as a bug if the algorithm's assumptions requires it to be sorted a certain way.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
Yeah, by implicit ordering, I mean an underspecified pipeline which has an undeclared dependency on some of the inputs being sorted. Unfortunately this is why the code in the cited paper worked on some systems and not others as the os.listdir() call didn't guarantee sorted results and thus the assumed invariants were broken.
Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
> How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
I once persuaded the dplyr maintainers not to do an update that might re-order rows after a filter(). I think the human tendency to think of database rows as existing in a fixed, given order, which will only be changed explicitly, is deep.
Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?
For example, polars internally hashes rows for some operations in ways that affect the ultimate ordering.
They do not guarantee stability of hashing algorithm or seed across versions and platforms.
(No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)
This behavior has repeatedly frustrated me. I am writing some new transformation, want to see the results, and my first few sentinel rows are nowhere to be seen because they have been shuffled.
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
Why not slap an order_by on the end of the query and you're all good? If that's expensive maybe you can add a limit clause to your source.
Sure, I can change what I get, I just disagree that my ordered table is not guaranteed to stay ordered.
Neither R nor Pandas will shuffle results by default.
Perhaps to get better results on benchmarks.
Or you know, just better performance for people that know how to use their tool of choice.
Wouldn't such people just pass in
1 reply →