Comment by jtbaker
13 hours ago
highest impact is when there is network overhead, like accessing them over S3/HTTP/S, and you're accessing many files as part of a query, like using a glob `-*.parquet` pattern. I imagine the same applies when you're doing local I/O, but not as pronounced of an improvement due to lower latency.
yes, makes sense because bandwidth is less likely to be the bottleneck