Of course, but the dataset was mostly strings that needed to be cross-referenced with GIS data. Tried every library under the sun. The greatest speed up I got was using polars to process the mostly-string CSVs, but didn't help much. With that said, I think polars was also just released when we were working with that dataset and I'm sure there's been a lot of performance improvements since then.
Of course, but the dataset was mostly strings that needed to be cross-referenced with GIS data. Tried every library under the sun. The greatest speed up I got was using polars to process the mostly-string CSVs, but didn't help much. With that said, I think polars was also just released when we were working with that dataset and I'm sure there's been a lot of performance improvements since then.