Comment by arromatic

I don't know what caused it but I suspected at the time, and still do, that it was simply business people getting more involved in order to drive growth.

The hostility was simply this. One day we had a dedicated high level Google engineer helping us out and giving us guidance (and even special tags) to get the data we needed in a cost effective manner for both Google and us. The next day, he was gone and we received demands to know exactly what we were doing, why and even sensitive information about our business. After several months of such probing, we were summarily told that the access we had was revoked and that there was no recourse.

We circumvented by setting up thousands of unique IP addresses in 50+ countries throughout the world and pointing our spiders at Google through them (same as they do to everyone else). These were throttled to maintain very low usage rates and stay off the radar. We continually refilled our queues with untouched IPs in case any were ever blacklisted (which happened occasionally).

As for what we did, we sampled ads for every keyword under the sun, aggregated and analyzed them to find out what was working and what wasn't. This even led to methods for estimating advertiser budgets. At one point, we had virtually every Google advertiser and their ongoing monthly spend, keywords and ad copy in our database. Highly valuable to smart marketers who were looking for an edge.