Comment by cobbzilla
3 years ago
Fantastic project.
Have you considered running (# of cpus) parallel scanners continuously? An inbound query “hops on” the the least-loaded scanner; at each article/chunk the scanner runs all the queries; each query “hops off” and returns after it has completed the cycle through the entire DB.
I do indeed use all 6 cores of my server simultaneously, and the caching on the database part of memory is quite effective. If I had a little spare cash I might just purchase some more CPUs :-)