Putting reverse indexes in S3 alongside large aggregate log objects has been an effective pattern for more than a decade. With objects in cheap access modes and indexes in modes that are cheaper to access it becomes far more cost effective than most options (cheap S3 clones aside).
The earliest code I have to do this looks to have been created in 2012.
Separation of compute and storage is such a fabulous pattern.
Well, it answers the technical questions of what changes when you stick your search indexes on slower and more expensive storage - but not why you would do that in the first place.
Appreciate the strategies that went into the work to optimize a high latency foundation to feasibly work as a search index.
Fundamentally, our work and stack goes against this specific architecture. Were I to make one, where scalability and efficiency for a search index is prioritized, I would approach it more from a regionally distributed hierarchical fractal architecture where caching is leveraged at every layer.
But architectures as presented in the original article put strong constraints on the solution space which is how exceptional insights and breakthroughs happen.
> On a local SSD, a random read costs nothing if a page is cached, and costs under a millisecond in case of a page fault. […] The scarcest resource is not CPU, and not bytes, but the number of round trips you make to the object store, and most design decisions below follow from that
There's not really a difference here between S3 and your disk, aside from the literal amount of latency. Restructuring the format to require less reads to make S3 faster would proportionally benefit disk reads.
S3 is a lot easier manage, though. Storage is just then AWS's problem. (And what's a few ms of latency on the BE when pages take tens of seconds to load on the whole.)
> What changes when your search index lives on S3?
None of this changes because your search index lives on S3; there's no property of S3 that says "this is an inherently better format for S3 than it is for disk". The format improvements are independent of the latency difference.
Putting reverse indexes in S3 alongside large aggregate log objects has been an effective pattern for more than a decade. With objects in cheap access modes and indexes in modes that are cheaper to access it becomes far more cost effective than most options (cheap S3 clones aside).
The earliest code I have to do this looks to have been created in 2012.
Separation of compute and storage is such a fabulous pattern.
Well, it answers the technical questions of what changes when you stick your search indexes on slower and more expensive storage - but not why you would do that in the first place.
it's web scale!
Mongo does this?
Isn't this the same as Elasticsearch searchable snapshots?
Reminds me of turbopuffer
Appreciate the strategies that went into the work to optimize a high latency foundation to feasibly work as a search index.
Fundamentally, our work and stack goes against this specific architecture. Were I to make one, where scalability and efficiency for a search index is prioritized, I would approach it more from a regionally distributed hierarchical fractal architecture where caching is leveraged at every layer.
But architectures as presented in the original article put strong constraints on the solution space which is how exceptional insights and breakthroughs happen.
You have my respects.
How about not putting it on S3?
> On a local SSD, a random read costs nothing if a page is cached, and costs under a millisecond in case of a page fault. […] The scarcest resource is not CPU, and not bytes, but the number of round trips you make to the object store, and most design decisions below follow from that
There's not really a difference here between S3 and your disk, aside from the literal amount of latency. Restructuring the format to require less reads to make S3 faster would proportionally benefit disk reads.
S3 is a lot easier manage, though. Storage is just then AWS's problem. (And what's a few ms of latency on the BE when pages take tens of seconds to load on the whole.)
> There's not really a difference here between S3 and your disk, aside from the literal amount of latency.
Latency is the whole point.
> Latency is the whole point.
The article's title is literally:
> What changes when your search index lives on S3?
None of this changes because your search index lives on S3; there's no property of S3 that says "this is an inherently better format for S3 than it is for disk". The format improvements are independent of the latency difference.
1 reply →