Comment by dekhn
3 hours ago
It's hard to take the article seriously when it has quotes like this: "The hardest part. 2B edges twitter graph is already huge (its edges are 30 GB in CSV !!!)."
Who cares how big the graph is in CSV? That's not the representation you operate over in big data.
All of this would have easily fit in memory on any reasonable modern system.
Here is a proposal built on parquet/arrow. What other alternatives exist that can be queried by a database without ingest?
https://github.com/Ladybug-Memory/icebug-format