Comment by marginalia_nu
3 years ago
Sometimes it's a lesser evil. Clouflare blocks about 1.6 million bot search queries per day on my search engine. Simply could not operate it without this inconvenience.
3 years ago
Sometimes it's a lesser evil. Clouflare blocks about 1.6 million bot search queries per day on my search engine. Simply could not operate it without this inconvenience.
> I'm currently looking for hosting for a large term frequency data file that is necessary for several of the search engine's core functions.
Did you get that sorted out?
Asking because we (sqlitebrowser.org, dbhub.io) have a bunch of Hetzner dedicated servers that are nowhere near fully utilised. Could probably figure something decent out using those, as Hetzner doesn't charge for bandwidth.
Yeah that's solved itself, I eventually got a cheap VPS @ downloads.marginalia.nu for providing these files. It solves the immediate problem of hosting the data that can't go in git.
How much space have you got by the way?
Checking just now, there's at least 1/2 TB spare on all except one machine.
For these boxes, once they're set up they tend to not grow all that much in disk space.
On a different note, is IPv6 supposed to be enabled for "downloads.marginalia.nu"?
1.6 million out of how many total?
50k legitimate queries / day on a slow day. A HN hug of death is maybe 100-150k/day.
I think those who haven't operated a publicly-visible server on the open Internet in some time might be surprised at just how shark-infested these waters are now. It's, like, mostly sharks.