Comment by devonkim

So basically you're talking about the design and limitations of https://databricks.com/blog/2015/04/13/deep-dive-into-spark-... I take it?

The implication of this work I thought is that it could be further expanded to other languages and DSLs. However, Spark's SQL generator still being very JVM-dependent and this optimizer generating bytecode kind of makes it pretty specific to JVM-supporting languages only. This would probably leave out Haskell and Erlang / Elixir in the short term, which is probably where I'd expect to see a different perspective on the whole data analytics front. We have Datomic, sure, (and I guess Clojure is enough for a lot of folks) but it'd be nice to have something other than "we want to make Hadoop... BUT BETTER" as a lot of the motivations.