Comment by procaryote

2 months ago

750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour

Millions per hour is tens per second though; perhaps the fix is performance improvements

3 comments

procaryote

Reply
setupminimal  2 months ago

We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.

andai  2 months ago

Yeah we can just rewrite the web servers in Rust ;)

That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!