Latest Register Log In

+ Advanced Search

Disallowing spiders from non-critical pages

Disallowing spiders from non-critical pages
10/23/06 (Edited 03/10/14)

At Admin Panel -> Settings -> SEO you have the option to block spiders from non-critical pages. This serves a dual purpose. First, it saves your server from some pounding it would otherwise take on irrelivent pages (like the "report post" page of each and every comment, the "email listing" page of every listing and the like). Secondly, it helps spiders get deeper into your site since they aren't drawn off into wild goose chases. If you're seeing 500 months of your site's empty calendar indexed while important pages aren't getting indexed, you may want to use this option.

"Non-critical" means everything except these pages:

  • any content from index.php (front page, category pages, custom templates)
  • anything on comments.php that doesn't have action= in it (in other words, the main thread view)
  • anything on link.php (listing details)
  • anything on memberlist.php (the member list and all member profiles)

URL rewriting is a non-factor, as the urls have been rewritten to their php forms before the determination is made. However, if you have URL rewriting enabled you may prefer to use the option to stop indexing of all non-rewritten URLs (which serves a similar purpose but is less strict, allows more to be indexed).

Description Details on the option.
Views 908 views. Averaging 0 views per day.