Comment by spratzt

The vast majority of businesses will never need more than single node architecture. Hardware advances are continually increasing that percentage.

SPARK and its modern counterpart Databricks are essentially obsolete for these organizations. Whatever justification they may have had in the past is no longer true.

I’ve recently closed down several in house SPARK clusters and replaced them with single nodes.

In addition to the simplicity of the design and reduction in cost there was a massive increase in performance. I expect this will become more common in the future; leaving distributed architecture for a small and increasingly niche group.