Long story short: Meilisearch is a very good search application for sure, but our needs drifted apart.
Longer version:
Our usecase isn't entirely aligned with how Meilisearch is meant to be used, I think. For one, we have a very high number of indices (at the time of using Meilisearch we had multiple per site that uses our service) and we wanted to have these replicated to edge nodes we have placed around the globe to keep latency down. In the end I was ending up spending an inordinate amount of time wrangling Meilisearch to get indices up and running across our different edge nodes. And, I am not sure if it was a regression in the last Meilisearch version we used, or user error, but at some point some index creations would just never get done, and would hold up our entire indexing pipeline. So long story short: the amount of effort needed to orchestrate and keep all the individual Meilisearch nodes up to date and in sync just turned out to be too much hassle.
The system I built afterwards works vastly better because it's actually bespoke and tailored to our use case. I can create the search indices on our crawler nodes entirely in isolation, run all sorts of preprocessing and optimizations once, and then just ship the final search indices to all the nodes that then serve them, and as a result I need wastly less computational resources on those edge nodes too, since no actual indexing work ever needs to be done on them.
It also means (double edges sword of course, because Meilisearch already gives you so much functionality out of the box) that I had to re-implement a whole lot of stuff around relevance tuning, snippet generation, ranking, etc, but as a result I have also been able to build interesting functionality than we didn't get out of the box before. And lastly, since the search index is now served as part of our main search node application things also get really fast, as I have no latency whatsoever to speak of going from app to Meilisearch.
Very interesting! I'm considering Meilisearch but have zero experience with it, so I appreciate your response, thanks. I've used Elastic in the past, but trying to avoid it now. I'm a ways from needing edge distribution, so hopefully Meili might be a decent choice until then. Isolated indexing shipped back to the core sounds pretty cool though.
Long story short: Meilisearch is a very good search application for sure, but our needs drifted apart.
Longer version: Our usecase isn't entirely aligned with how Meilisearch is meant to be used, I think. For one, we have a very high number of indices (at the time of using Meilisearch we had multiple per site that uses our service) and we wanted to have these replicated to edge nodes we have placed around the globe to keep latency down. In the end I was ending up spending an inordinate amount of time wrangling Meilisearch to get indices up and running across our different edge nodes. And, I am not sure if it was a regression in the last Meilisearch version we used, or user error, but at some point some index creations would just never get done, and would hold up our entire indexing pipeline. So long story short: the amount of effort needed to orchestrate and keep all the individual Meilisearch nodes up to date and in sync just turned out to be too much hassle.
The system I built afterwards works vastly better because it's actually bespoke and tailored to our use case. I can create the search indices on our crawler nodes entirely in isolation, run all sorts of preprocessing and optimizations once, and then just ship the final search indices to all the nodes that then serve them, and as a result I need wastly less computational resources on those edge nodes too, since no actual indexing work ever needs to be done on them.
It also means (double edges sword of course, because Meilisearch already gives you so much functionality out of the box) that I had to re-implement a whole lot of stuff around relevance tuning, snippet generation, ranking, etc, but as a result I have also been able to build interesting functionality than we didn't get out of the box before. And lastly, since the search index is now served as part of our main search node application things also get really fast, as I have no latency whatsoever to speak of going from app to Meilisearch.
Very interesting! I'm considering Meilisearch but have zero experience with it, so I appreciate your response, thanks. I've used Elastic in the past, but trying to avoid it now. I'm a ways from needing edge distribution, so hopefully Meili might be a decent choice until then. Isolated indexing shipped back to the core sounds pretty cool though.