Comment by embedding-shape
20 days ago
> I rewrote some Bash and Python scripts into C# that were previously processing gigabytes of JSON
Hah, incredibly funny, I remember doing the complete opposite about 15 years ago, some beginner developer had setup a whole interconnected system with multiple processes and what not in order to process a bunch of JSON and it took forever. Got replaced with a bash script + Python!
> Just how much data do you need when these sort of clustered approaches really start to make sense?
I dunno exactly what thresholds others use, but I usually say if it'd take longer than a day to process (efficiently), then you probably want to figure out a better way than just running a program on a single machine to do it.
A C# executable is more similar to a bash/Python script than it is to a interconnected system using multiple processes.
Yeah, I realize now that my comment actually misses the most important point, the "interconnected system with multiple processes" I was talking about was made in C#, that's why the whole "I did the reverse as you" was funny to me in the first place.