Comment by jancona

Bret wrote: "I think it was bad for us to stray from Internet standards and conventions by having an robots.txt that was open and a separate agreement with additional restrictions."

You don't have an "agreement" at all. An agreement requires that two parties actually, um, agree. You've published a statement where you assert certain rights and imply that you will sue anyone who accesses data on your site in a way you don't like. You may get away with that, regardless of the legal merits of your position, because you have more money for lawyers than most people you're likely to sue. But don't try to dignify what you're doing by calling it an "agreement". It's like an extortionist telling me that we have an agreement that he won't break my windows if I pay him protection money.