Comment by 0zymandiass
5 years ago
Reminds me of https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Cluster computing can be useful, but until you're talking about petabytes of data, it probably isn't helping you
5 years ago
Reminds me of https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Cluster computing can be useful, but until you're talking about petabytes of data, it probably isn't helping you
Kind of reminds me of the early days of like, Hadoop, when it was all the rage. Then people realized they could do most that stuff in a python script on a single machine in less time because of all the bookkeeping overhead and complexity.
Today they still do it in python except it’s now running on Spark across sprawling expensive clusters to do complex tasks like transform text or coalesce two fields.