[hunchentoot-devel] googlebot revisitation rate excessive?

[Jeff Cunningham]

Zach Beane wrote:
> On Sun, Jun 29, 2008 at 04:31:47PM -0700, Jeff Cunningham wrote:
>   
>> Has anyone else noticed excessive visitation of pages served dynamically 
>> by Hunchentoot? On a couple of small sites virtually every page gets hit 
>> every five minutes by http://www.google.com/bot.html, regular as clockwork.
>>
>> Normally, I setup robots.txt to allow google to walk these sites but 
>> unless I can think of a way to tell them to back off to once a week or 
>> something I'm about to Disallow them. They're just little private sites 
>> and Google ends up being 99% of the traffic. It's silly.
>>     
>
> FWIW, I haven't had any trouble like that with two different
> Hunchentoot sites.
>   
Is there something you do to indicate that the page has a fixed date? I 
was wondering if maybe the fact that dynamically generated pages look 
"changed" to Google, so they get thrown back in the queue to be rechecked.