Comment by panzi
5 days ago
Interesting!
We're currently use pgvectorscale. Some questions:
- Will pg_vectorsearch eventually also support vectors bigger than 1968 dimensions? E.g. VoyageAI can create embeddings with 2048 dimensions (but we don't actually need that at the moment, only 1024).
- Why did you choose C this time and not Rust, as you did with pgvectorscale?
- Why did you not just extend pgvectorscale with another index type instead of creating a whole new extensions?
- Almost certainly. In fact, there is already a submitted PR contribution for larger dimensions.
- C is much better integrated into Postgres, doesn't require pgrx. Given also experience developing pg_textsearch, and the acceleration from AI-assisted development and testing, have moved back to more C with extensions.
- Two reasons. First, decision to move back to C. Second, if there ultimately isn't much code overlap, better to keep them separate for easier adoption (smaller footprint and security/correctness envelop)