Comment by x0xMaximus

We run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool.

While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.