Comment by derefr
4 days ago
It's one of those paradoxes inherent to software development: even if a tool is designed to work with Big Data, developers are too impatient / feel too unproductive to actually work with Big Data during development, so they only bother testing their tool using small data for 99.9% of development. Problems the tool has under actual Big Data tend to only start being found as of v1.0-rc1.
it's also an inherit bash problem where what you write is what you get, there's no buffer to keep you from slowing something down to a crawl.