Comment by addisonj
2 years ago
Congrats on the launch!
I have quite a few years of observability experience behind me and hand't really considered some of the unique aspects that LLMs bring into the picture. Here are a few thoughts, responses to your questions, and feedback items
* Generally, I think you do a good job of having a clear, concise story and value proposition that is fairly early in a market where the number of people hitting these problems is rapidly growing, which is a pretty nice place to be! But, I do think that can be a challenge in that you have to help people recognize the problem, which often means lots of content and lots of outreach.
* I think going open-source and following a PLG model of cloud/managed services is pretty reasonable way to go and certainly can be a leg up over the existing players, but I noticed in your pricing a note about enterprise support of self-hosting in customer VPC and dedicated instances. There is lots of money there... but it also can just be extremely big time sink for early stage teams, so I would be careful, or at least make sure you price it such that it supports hiring.
* Also on pricing, I wonder if doing this based on storage is how people would think about? Generally, I think about observability data in terms of events/sec first and then retention period. If you can make it work with a single usage based metric of storage, than that is great! but I would be concerned that 1) you aren't telling the user which plan can support throughput and 2) you could end up with some large variance in cost based on different usage patterns
* The biggest question I have is how much did you explore opentelemetry? Obviously, it is not as simple as just going and building your own API and SDK... but when I look at the capabilities, I could see opentelemetry being the underlying protocol with some thinner convenience wrappers on top. From your other comments, I understand that you see some ways in which this data is different than typical trace/observability data, but I do wonder if that choice will 1) scare off some companies that are already "all in" on otel and 2) you don't get any opportunity to use all of the stuff around otel, for example, Kafka integration if you someday need that.
* As far as your question about OLAP, I wouldn't rush it... In general, once you are big enough that the cost/scalability limitations of PG are looming, you will be a different company and know a lot more about the real requirements. I will also say that in all likelihood, ClickHouse is probably the right choice, but even knowing that, there are lots of different ways to tackle that problem (like using hosted vs self-managed) and the right way to do it will depend on usage patterns, cost structure, where you end up with enterprise dedicated / self-hosted, etc. I will mention though that timescaledb is not a bad way to maybe buy you a bit of headroom, but it is important to note that the timescaledb offered by supabase shouldn't be compared to timescaledb community / cloud. The supabase version isn't bad, it just isn't quite the same thing (i.e. no horizontal scalability)
Anyways, congrats again! It looks like you are off to a good start.
If you have any other questions for me, my email is in my profile.
+1 on the OTel mention. Having telemetry in place in a system, one would typically implement a single behavioral tracking SDK on top. Adding yet another SDK for LLMs is a hard ask given how specific the implementation will be. Backing back on a standard you offer value-added insights on top.
On the other hand, if you target just the applications that implement an API behind an LLM, you will have customers expecting value-added services on top of telemetry, like prompt optimization, classification, result caching, etc.
Your choice which direction and target group you will focus on first.
This reads like a book, thank you so much for putting this together!
> About value prop: Thanks for the feedback! We are already trying to be as vocal about it as possible by writing great docs etc. but can probably do better.
> PLG & OSS: thanks for the hint, we will be careful around managing deployments within customer VPCs.
> Pricing: Currently picked storage as the first metric to price on as this varies a lot across users. Some use langfuse to track complex embedding processes with a lot of context, others just simple chat messages with relatively low-context, low-value events.
> OTel: We looked into it but did not go into all the details. We wanted to have a product out there fast and liked the experience of e.g. Posthog SDKs. I might reach out to you concerning this topic after investing more time on it. Thanks for the offer!
> OLAP: Agree, i also learned to tackle scaling issues once they appear and so far we are good. Interesting that Supabase has no horizontal scaling. This would be one of the main reasons to use it IMO.
Re: Supabase and timescaledb.
Just want to make a bit more clear, Supabase has the ability to do some distribution via replication, but it isn't a true multi-master DB.
Timescaledb does support a multi-node config (https://docs.timescale.com/self-hosted/latest/multinode-time...) on top of postgres but that isn't in the open-source apache-licensed version, instead it is only in Timescales's community BSL version which isn't license compatible with supabase
And yeah, please don't hesitate to reach out in regards to OTel... lots of opportunity but also not as simple ;)
This reply is top notch. I agree with the points here. Definitely worth considering this advice.