PEAK XOOPS - Googlebot vs piCal 
XOOPS
XOOPS : Googlebot vs piCal
Recently, the algorithm of Googlebot changed. (This is just a guess of myself)
And now, it accesses piCal's links very frequently.
As you know, piCal has very many internal links.
If Googlebot crawls is going to access all of piCal's links, it will be high load of the server.
Thus I've just modified piCal as outputing meta header like this:
<meta name="robots" content="index,nofollow" />
If this modification is no use for your site, check your theme.html this meta header exists or not.
<meta name="robots" content="<{$xoops_meta_robots}>" />
Or, you can edit robots.txt like this:
User-agent: *
Disallow: /(xoops)/cgi-bin/
Disallow: /(xoops)/tmp/
Disallow: /(xoops)/cache/
Disallow: /(xoops)/class/
Disallow: /(xoops)/images/
Disallow: /(xoops)/include/
Disallow: /(xoops)/install/
Disallow: /(xoops)/kernel/
Disallow: /(xoops)/language/
Disallow: /(xoops)/templates_c/
Disallow: /(xoops)/themes/
Disallow: /(xoops)/uploads/
Disallow: /(xoops)/modules/piCal/
But, the method of meta header is more desirable, I think.
Comments list
edwina
Posted on 2009/2/2 21:49
Googlebot, grandma called and said you have to go home because your ruining everyones life and eating all her bandwidth.there is actually an error in this calendar module I have installed in Xoops on the Pikes website. It seems that the piCal module is have thousands of random URL's indexed by our friend google. I dont know what the heck it's indexing but it's taking up 5K every time it does, and now we are only 10 days into the month of feruary and already pushing into the 1 gig bandwidth limit.
-------------------------------
edwina
<a href="http://mls.fastrealestate.net/b/flat-fee-mls-2">Flat Fee MLS</a>
-------------------------------
edwina
<a href="http://mls.fastrealestate.net/b/flat-fee-mls-2">Flat Fee MLS</a>
GIJOE
Posted on 2007/6/7 6:06
Quote:You should point which is the problem.
<{$xoops_meta_robots}> from Preferences can be overrided by piCal.
(You can see this site)
Is your report the trouble that <{$xoops_meta_robots}> from Preferences cannot be overrided by piCal?
If so, tell me the core version of your XOOPS.
The theme I've been using has the line:
<meta name="robots" content="<{$xoops_meta_robots}>" />
..And I'm using the latest piCal index.php which attempts to assign the "nofollow" directive:
$xoopsTpl->assign( "xoops_meta_robots" , $meta_robots ) ;
But for some reason the page is compiled with the Robots rule as set by the main Xoops preferences. For now I'm using "Disallow" but would rather have "nofollow" so more of the calendar gets indexed.
<{$xoops_meta_robots}> from Preferences can be overrided by piCal.
(You can see this site)
Is your report the trouble that <{$xoops_meta_robots}> from Preferences cannot be overrided by piCal?
If so, tell me the core version of your XOOPS.
swingstylez
Posted on 2007/6/6 18:36
Hello GIJoe,
Thanks for you development - the calendar was just what I needed for my hobby site and is the main feature :).
The theme I've been using has the line:
<meta name="robots" content="<{$xoops_meta_robots}>" />
..And I'm using the latest piCal index.php which attempts to assign the "nofollow" directive:
$xoopsTpl->assign( "xoops_meta_robots" , $meta_robots ) ;
But for some reason the page is compiled with the Robots rule as set by the main Xoops preferences. For now I'm using "Disallow" but would rather have "nofollow" so more of the calendar gets indexed.
Also, the other reason your calendar is so great is because you post calendar events with their category names as style ids, allowing me to colours code my calendar. Please keep doing it like that :)
Regards,
Ben
www.swingjunction.co.uk
Thanks for you development - the calendar was just what I needed for my hobby site and is the main feature :).
The theme I've been using has the line:
<meta name="robots" content="<{$xoops_meta_robots}>" />
..And I'm using the latest piCal index.php which attempts to assign the "nofollow" directive:
$xoopsTpl->assign( "xoops_meta_robots" , $meta_robots ) ;
But for some reason the page is compiled with the Robots rule as set by the main Xoops preferences. For now I'm using "Disallow" but would rather have "nofollow" so more of the calendar gets indexed.
Also, the other reason your calendar is so great is because you post calendar events with their category names as style ids, allowing me to colours code my calendar. Please keep doing it like that :)
Regards,
Ben
www.swingjunction.co.uk
JasonMR
Posted on 2004/11/11 21:39
I've also noticed more googlebots than usual, sniffing around piCal.
Thanks for the fine explanation and update!
Thanks for the fine explanation and update!
Login