Latest Register Log In

+ Advanced Search

Disallowing spiders from non-critical pages

Disallowing spiders from non-critical pages
10/23/06 (Edited 03/10/14)

At Admin Panel -> Settings -> SEO you have the option to block spiders from non-critical pages. This serves a dual purpose. First, it saves your server from some pounding it would otherwise take on irrelivent pages (like the "report post" page of each and every post, the "email topic" page of every topic and the like). Secondly, it helps spiders get deeper into your site since they aren't drawn off into wild goose chases. If you're seeing 500 months of your site's empty calendar indexed while important pages aren't getting indexed, you may want to use this option.

"Non-critical" means everything except these pages:

  • any content from index.php (front page, forum pages, custom templates)
  • anything on posts.php that doesn't have action= in it (in other words, the main thread view)
  • anything on link.php (topic details)
  • anything on memberlist.php (the member list and all member profiles)

URL rewriting is a non-factor, as the urls have been rewritten to their php forms before the determination is made. However, if you have URL rewriting enabled you may prefer to use the option to stop indexing of all non-rewritten URLs (which serves a similar purpose but is less strict, allows more to be indexed).

Description Details on the option.
Views 908 views. Averaging 0 views per day.