Comment by perrygeo
3 days ago
Polars is much better than pandas in this regard. But using it via Python still carries the risk of runtime errors.
Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
When I deal with datascience, I rarely have type problems in my programs that python typecheckers can't catch. I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
> I rarely have type problems in my programs that python typecheckers can't catch.
> I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
Something about those two statements makes my brain explode. You rarely have type problems, except when dealing with data? Isn't the whole point of a program to deal with data?
If it’s 100% type checked and you’re still hitting massive numbers of type related runtime errors, it probably isn’t 100% checked.
Empirically false. Python's type system is not sufficient to describe runtime variants, period. I have worked on at least three system that used mypy in strict mode yet hit runtime errors constantly.
I get where you're coming from. A good type system should provide invariant guarantees about runtime behavior. But Python's does not.