Neki – Sharded Postgres

18 hours ago (planetscale.com)

Folks, in your launch posts remember to describe what you're launching and what it's for. Ideally in the opening paragraph.

- Intro: Explains problem but not what is Neki or what it's for.

- Why Neki: Explains alternatives and why they suck but not what is Neki or what it's for.

- How does Neki work: Describes the technical components but not what is Neki or what it's for.

- What you get beyond sharding: Explains how to run/deploy but not what is Neki or what it's for.

- What is a platform preview + Try Neki today: 140 words on the definition of "preview" and links to get started... but not what is Neki or what it's for.

Edit: The landing page has 100x more useful information up front -> https://neki.dev/

Edit #2: They've since added a "What is Neki" section. Good reaction time.

  • I scrolled through the landing page and even the "Get started" page and I have still no idea what is this. Is this a managed db? Something I can run myself? No idea.

  • Folks, your blog post sucks at marketing.

    Edit: your marketing site actually has excellent marketing information.

    Edit #2: They've since replaced the blog post with a link to the marketing page. Good reaction time.

  • The landing page which is linked at the start of the second sentence of this page…? I don’t completely disagree with you, but this is mostly hypertext working as intended, is it not?

    • Yeah this meta commentary on HN is very confusing for me. The structure of the article is literally

      - short paragraph saying "we launched neki"

      - short paragraph saying why neki was needed

      - paragraphs about what neki is (with a header)

      That seems like a very reasonable structure for a blog post like this one. Not being able to get to the third paragraph of a blog post seems like a "you" problem, not a blog problem.

      1 reply →

I love the offer which is running on the "PlanetScale Metal".

They claim to provide unlimited IOPS (I/O operations Per Second).

I would like to get some of these drives myself. Sounds like a technical miracle.

On the other hand claiming such miracles makes me wonder which other aspects in this are are actually a working miracle or rather a mythos/false marketing claim.

The top concern I've gotten from dev teams when proposing HA distributed postgres (e.g through RDS Aurora global) is that eventual consistency is not suitable for many workloads.

Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.

  • Assuming it works the same way as their MySQL product, Vitess:

    The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be.

    I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)

  • Same here, first thing I need to know when considering a distributed system is how consistency is handled. If it's eventual consistency, what is the replication lag like? If it's strong consistency, can their network handle that? What happens when a node goes down?

I love how the Planetscale CEO has been shitting on multigres[0], Supabase's equivalent to Neki, and gloating about how superior Neki is, and yet Neki is not even open source!

Sam, it feels like you're just playing games. It's hard to take anything you say or do seriously. Are you going to OSS this thing? Or have you decided to backpedal?

Congrats on the checks notes closed launch, I guess?

0: https://github.com/multigres/multigres

  • How does open source have anything to do with Neki's ability to scale against a service that doesn't even do sharding.

    • They both are characteristics of software that people consider when deciding whether to use it.

      Open source being by far the more important of the two, since it means the user is more insulated by terrible corporate decisions thus making it more suitable for long term planning.

      13 replies →

  • Do they get the same backlash if they went with a source available license similar to MongoDB or Elastic?

Kinda ironic that Planetscale built their company on open source Vitess that google built

And now planetscale have made a Vitess like equivalent for Postgres which they have made proprietary

  • They have made it proprietary so that providers like AWS can't build their own service with it, thus eating into PS's income.

    Google is big enough to survive open-sourcing Vitess, k8s, angular, etc. Planetscale isn't.

    • correct. AWS do anti-competitive stuff to us all the time we can't make it any easier for them.

    • Google, Azure, and AWS already have their highly available PostgreSQL alternatives.

I need this for my side project that has zero users, but I wonder if Neki will scale well enough.

I love to see this but as a heavy Vitess/MySQL user I do fear the split focus from PlanetScale. Hoping to see continued improvements on the Vitess side as well.

Selfish doubts aside, congrats to Planetscale on the launch!

How are cross-shard joins and transactions handled? Are there query patterns that would require application changes despite using the same drivers and ORM?

Is this going to be open sourced? I thought this was the plan at the start. But ow I am confused.

I am on a 12 Muni right now that is plastered with a Neki ad, and I looked up neki.dev literally 10 mins ago to see what this product is about. Did they time the release of this bus ad with the launch date?

  • Bus ads have been running for a little while now. A few billboards around too.

Neki is a sharded Postgres implementation. It introduces one level of indirection between the application and the Postgres shards.

Found this from neki.dev.

So apparently the neki router uses many parts of Postgres. But is it open source itself?

Apparently not.

A C2 level diagram included in the blog post would be helpful. What are the drawbacks to using Neki aside from the obvious (new software has bugs)?

  • The first thing I want to know when looking at a distributed system is how consistency is handled, ctrl+f for "consis" on both the blog post and neki.dev has zero hits. So I can only assume it will have eventual consistency with a considerable replica lag for real-time systems.

Pretty exciting to see a sharded Postgres product in the market.

The pitch is compelling. I wonder how many teams will be able to operate sharded database setups in production as a result of this.

Obviously welcome more tools for Postgres and sharding but it’s a tough hill to climb against the hyper clouds. Best wishes to the team.

  • The largest database to ever run on Aurora MySQL runs on PlanetScale, and the largest database to ever run on Aurora limitless also runs on PlanetScale.

  • We went as far as we could on GCP before PlanetScale. Spanner is amazing but also amazingly expensive. And CloudSQL, also amazing as long as you don’t need write scaling. PlanetScale (Vitess+MySQL plus their branching / deployment and monitoring tools) is just not a combination offered on GCP. And while I didn’t use Aurora, from talking to and exploring AWS it doesn’t really have this combination either.

I see no mention of foreign keys, or any other constraints, across shards. If, as I suspect, they're not implemented, it would still be useful but at the level of Spanner 10 years ago.

Cool, but we're on MySQL - anything similar available? Percona XtraDB?

  • That is what the “regular” PlanetScale is all about. It uses the Vitess open source project to do write scaling through shading. It is very powerful and customizable and stable. It was built to run YouTube. Former happy user here.

only planet scale? Rookie numbers.

I am looking for interdimensional and interuniversal scale. Which of you trust fund babies has a startup which is working on this problem?

The tech might be amazing, but the CEO comes across as an asshole