Hb_Kai Posted February 3, 2009 Report Share Posted February 3, 2009 I've finally set up a new and much more improved Section32, SECT32, but I'm having some problems finding it on Google.I understand that the spiders search day in and day out new websites for it to crawl but my website is protected by a log in page so that no guests can view the content. I'm thinking about changing this though so that they can view but not post.But anyway, how do I give the search engines' spiders the authorisation to crawl my forum? Quote Link to comment Share on other sites More sharing options...
ɹəuəllıʍ ʇɐb Posted February 4, 2009 Report Share Posted February 4, 2009 By default, permission is always granted. To alter the default permission, use a robots.txt file. Quote Link to comment Share on other sites More sharing options...
Hb_Kai Posted February 4, 2009 Author Report Share Posted February 4, 2009 What about password protected forums? Quote Link to comment Share on other sites More sharing options...
ɹəuəllıʍ ʇɐb Posted February 4, 2009 Report Share Posted February 4, 2009 Naturally, password protected forums, or password protected forum sections cannot be spidered by robots. Quote Link to comment Share on other sites More sharing options...
Hb_Kai Posted February 4, 2009 Author Report Share Posted February 4, 2009 Is there no setting to allow authorisation by creating a login for spiders? Quote Link to comment Share on other sites More sharing options...
ɹəuəllıʍ ʇɐb Posted February 5, 2009 Report Share Posted February 5, 2009 Actually I don't know the details about that. As you can see at the bottom of this forum, we have a couple of spiders (Google.com, Inktomi Spider) that seem to be logged in. You would have to ask Craig or Chris how that works. Or maybe check the documentation on the Invision website. Quote Link to comment Share on other sites More sharing options...
Hb_Kai Posted February 5, 2009 Author Report Share Posted February 5, 2009 Hmm. I don't know Craig, but I've seen Chris around. I think I'll go ask him. Or would it be wasting his time sending him a PM? Quote Link to comment Share on other sites More sharing options...
ɹəuəllıʍ ʇɐb Posted February 6, 2009 Report Share Posted February 6, 2009 Chris is probably the best person to ask; try PM him. Quote Link to comment Share on other sites More sharing options...
Hb_Kai Posted February 6, 2009 Author Report Share Posted February 6, 2009 Thanks Pat.I've PM'ed him so I'll wait for a reply. :) Quote Link to comment Share on other sites More sharing options...
scrum Posted March 2, 2009 Report Share Posted March 2, 2009 By default, permission is always granted. To alter the default permission, use a robots.txt file.The key is in having a proper robots.txt file. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.