Comment by immibis
2 months ago
The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
2 months ago
The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.
"Making a search engine is unethical" is certainly one of the takes of all time. I'm sure Google is glad you believe it.