Comment by winniewinnie
3 days ago
By "implicit ordering", do you mean "implicitly assumed that the data is ordered a certain way"? Since if that assumption of data being sorted a certain way is broken on some systems and not others, the result might be both non-deterministic (which could be a bug if the result is not allowed to be non-deterministic, but may or may not be a bug regarding the algorithm's assumptions) as well as a bug if the algorithm's assumptions requires it to be sorted a certain way.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
Yeah, by implicit ordering, I mean an underspecified pipeline which has an undeclared dependency on some of the inputs being sorted. Unfortunately this is why the code in the cited paper worked on some systems and not others as the os.listdir() call didn't guarantee sorted results and thus the assumed invariants were broken.
Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
> How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.