Comment by blancotech
3 days ago
Curious, how are you keeping the product data up-to-date? We built something similar for price alerts on specific URLs, that we use all the time, but have to poll it daily to see the price change (https://lowlow.bot). I imagine that would be a lot of $$ for every product on the Internet.
This is definitely one of our hard problems. There are some optimizations -- e-tags / last modified headers, comparing page content hashes -- but there's also only so much you can do before you just have to check the page again.
Last modified headers is always set to the current time in the majority of cases, and it also requires a web request too (albeit a HEAD request would likely suffice)