Comment by bormaj
6 days ago
Exciting project and definitely something I'd like to explore using. I particularly like the look of the API ergonomics. A few questions:
- is the schema inferred from the data? - can/does the schema evolve? - are custom partitions supported? - is there a roadmap for future features?
Thanks! Let’s go by parts, as Jason would say
Schema inference: yes, Arc infers the schema automatically from incoming data (both for MessagePack and Line Protocol). Each measurement becomes a table, and fields/tags map to columns.
Schema evolution: supported. New fields can appear at any time, they’re added to the Parquet schema automatically without migration or downtime.
Custom partitions: currently partitioning is time-based (hour-level by default), but custom partitioning by tag or host or whatever is planned. The idea is to allow you to group by any tag (e.g. device, region) in the storage path for large-scale IoT data.
Roadmap: absolutely. Grafana data source, Prometheus remote write, retention policies, gRPC streaming, and distributed query execution are all in the works.
We are going to start to blogging about it, so, stay tune.
Would love any feedback on what you’d prioritize or what would make adoption easier for your use case.
My use case isn't IOT, but about once a month I get a massive data dump from a vendor. Think tens of millions of rows and 100+ columns. Cleaning, ingesting and querying this data via standard RDBMS is a slow and brittle process. There is a time series aspect, but partitioning across other keys/groups is critical.