Googlebot vs piCal
Date 2004-11-09 18:57:57 | Category: XOOPS
|
Recently, the algorithm of Googlebot changed. (This is just a guess of myself)
And now, it accesses piCal's links very frequently.
As you know, piCal has very many internal links.
If Googlebot crawls is going to access all of piCal's links, it will be high load of the server.
Thus I've just modified piCal as outputing meta header like this:
<meta name="robots" content="index,nofollow" />
If this modification is no use for your site, check your theme.html this meta header exists or not.
<meta name="robots" content="<{$xoops_meta_robots}>" />
Or, you can edit robots.txt like this:
User-agent: *
Disallow: /(xoops)/cgi-bin/
Disallow: /(xoops)/tmp/
Disallow: /(xoops)/cache/
Disallow: /(xoops)/class/
Disallow: /(xoops)/images/
Disallow: /(xoops)/include/
Disallow: /(xoops)/install/
Disallow: /(xoops)/kernel/
Disallow: /(xoops)/language/
Disallow: /(xoops)/templates_c/
Disallow: /(xoops)/themes/
Disallow: /(xoops)/uploads/
Disallow: /(xoops)/modules/piCal/
But, the method of meta header is more desirable, I think.
|
|