Your Cart

Shopping Cart Items: 0   Sub-Total : $0.00    View Cart    Checkout
Welcome, Guest ( Customer Panel | Login )


Home | Profile | Register | Active Topics | Members | Search | FAQ


Forgot your Password?
 All Forums
 VPCart Forum
 General help me questions
 sample robots.txt file?		  New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

Dubbya
Starting Member

Canada
30 Posts
Posted - August 21 2006 :  08:58:31  Show Profile  Visit Dubbya's Homepage  Reply with Quote
Hi,

I'm looking for a good robots.txt file that lists the files we don't want indexed, cached or archived by including a list of email harvestors.

Obviously, there are key pages we do want indexed but there are a ton of files and some folders that should not be indexed.

I'm thinking this should be a standard file to be included with all versions of vp-asp.




Thoughts, comments, suggestions?

Always remember that wherever you go, there you are.

gupster
VP-CART New User

101 Posts
Posted - August 21 2006 :  14:16:18  Show Profile  Visit gupster's Homepage  Reply with Quote
Ive got the .txt off my site you can have if you want, the robots.txt files are ok but the dodgy listers will use this info a make sure they list what is on it! I use mine to stop google and yahoo listing my hundreds of pictures etc and using all my bandwidth!

# robots.txt for http://www.reactionclubwear.co.uk/

User-agent:	*
Disallow:	/cgi-bin
Disallow:	/images
Disallow:	/editor
Disallow:	/export
Disallow:	/import
Disallow:	/lang
Disallow:	/stylesheets


www.reactionclubwear.co.uk
www.reactionclothing.co.uk
www.reactionlingerie.co.uk
www.reactionvision.co.uk
Go to Top of Page

Dubbya
Starting Member

Canada
30 Posts
Posted - August 21 2006 :  15:33:24  Show Profile  Visit Dubbya's Homepage  Reply with Quote
Thanks gupster,

I was looking for an example of which individual files shouldn't be indexed. For example, the admin screens, login pages, customer forms, any of the files used during the ordering process, shop and customer email pages, that sort of thing.

I just don't want to have to do it all from scratch if I don't have to.

Always remember that wherever you go, there you are.
Go to Top of Page

Dubbya
Starting Member

Canada
30 Posts
Posted - August 23 2006 :  13:10:25  Show Profile  Visit Dubbya's Homepage  Reply with Quote
Here's what I've come up with. Since I've put my store in the root of my web, it's a little different.


User-agent: *
Disallow: /cgi-bin/
Disallow: /_borders/
Disallow: /_derived/
Disallow: /_fpclass/
Disallow: /_overlay/
Disallow: /_private/
Disallow: /_themes/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_map/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /aspnet_client/
Disallow: /images/
Disallow: /scripts/
Disallow: /stylesheets/
Disallow: /global
Disallow: /paypal
Disallow: /shop
Disallow: /ssl
Disallow: /tmp_
Disallow: /update_
Disallow: /ups
Disallow: /vs
Disallow: /yellow_


User-agent: Googlebot-Image
Disallow: /


This should keep bots from indexing any store scripts they don't need to while still spidering any other content.

If anyone is running a FrontPage web, It's wise not to let the FrontPage _vti folders get indexed either.

Alternatively, you can put the following meta tag <TITLE> element of each file you don't want indexed such as the store admin pages.

<meta name="robots" content="noindex, nofollow">


If anyone sees any mistakes, or knows of a better way to do this, please let us know.

Always remember that wherever you go, there you are.
Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
Snitz Forums 2000
0 Item(s)
$0.00
  • Your Cart Is Empty