Comment by andai
10 days ago
Interesting. How much does it cost to build a search index in current year? Also, how do you get around the fact that everyone's blocking bots?
I've been thinking of making a "small web" indexer so I'm curious about that. I'm seeing even tiny websites being behind CloudFlare, Anubis etc. these days. (And everyone complaining they're getting hammered by mysterious distributed HTTP traffic!)
Most of the cost is storage (database, indexes) and memory (RAM, search), and most of the risk is crawlers (they fail, they're reported as abuse, etc.).
You can look for Common Crawl or Open Web Index for dataset sizes and how many URLs those include to get a sense of baseline storage costs, and then 2x that for minimum usability.
It's honestly a bit tough to accept that we got a report of abuse and are still dealing with the aftermath of that after having a single crawler go haywire for a few hours (because we play nice and identify ourselves properly), but these... "mysterious" bots that keep hitting all the servers everywhere thousands of times per day just go on like nothing's happening and "no one"'s to blame.
I have been working on an indexer for myself using data from common crawl. They do all the leg work and just host the outcome.
Common Crawl is a really cool project! We're looking to implement a similar EU-based version: https://openwebindex.eu