Comment by wood_spirit
1 year ago
Quick for building a website? Probably not.
Quick for evaluating some idea you just had if you are a quant? Yes absolutely!
So imagine you have a massive dataset, and an idea.
For testing out your idea you want that data to be in an “online analytical processing” (OLAP) kind of database. These typically store the data by column not row and other tricks to speed up crunching through reads, trading off write performance etc.
There are several big tech choices you could make. Mainstream king is SQL.
Something that was trendy a few years ago in the nosql revolution was to write some scala at the repl.
It is these that K is competing with, and being faster than.
I would probably use Matlab for that sort of stuff tbh. Is K faster than Matlab?
For using a heap of libs to compute something that has been done a thousand times? No. For writing a completely new data processing pipeline from scratch? Much, much faster! Array langs have decent performance, but that's not why they are used. They are used because you develop faster for the kind of problem they're good at. People always conflate those two aspects. I use J for scientific research.
Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims:
> real-sql(k) is consistently 100 times faster (or more) than redshift, bigquery, snowflake, spark, mongodb, postgres, ..
> same data. same queries. same hardware. anyone can run the scripts.
7 replies →
Yeah to be clear I meant performance-wise. In terms of development speed it looks like it just goes to insane extreme on the "fast at the beginning / fast in the middle" trade-off. You know how dynamic typing can be faster to develop when you're writing a one-man 100 line script, but once you get beyond that the extra effort to add static types means you overtake dynamic typing.
1 reply →
I think some things will be faster and the real difference is more in things that are easy to express in the language. Eg solving differential equations may be easier in matlab and as-of joins in k.