Return to Help Center

Welcome to the Bravenet Help Forums. Here you can chat and meet other Bravenet Members. Please do not post advertisements.

Forum: Web Hosting Discussions
This forum is locked and posting is not allowed
Author
Comment
How do you allow or disallow web crawlers on your site

Been reading for hours it has to do with a robots.txt file, but I am totally lost.

Messed with the setting a few months back and now fear that the google crawler may have been blocked in my settings.

Was getting about 400 hits daily and now only about 150.

Re: How do you allow or disallow web crawlers on your site

There could be all kinds of reasons for the drop in hits. But I had no trouble finding your site in Google. Even a search for text in the latest post in your forum worked.

Re: How do you allow or disallow web crawlers on your site

Sign up for Google's Webmaster Tools. They will give you a one-line authentication meta code called "google-site-verification" to place in your site's page code in the head section that will make your website more Google friendly; that is, they will defintely know you exist and send their robots to index your site regularly.

I write my own code to blank pages, so don't ask me how to insert the one line code if you are using a template. I'm sure it's possible however.

Regardless, Google finds your site because it is linked from other sites or you submitted the link directly to them (through Webmaster Tools). If other sites out there dropped their link to your site that could result in less traffic. Or, other new sites have been added to their index that rank higher than yours when people do a keyword search.

Robots.txt is mainly used to exclude certain folders or file types from inclusion in search engines, or to keep certain search engines or directories from accessing your site. Your robots.txt file is set up to allow all search engines to access your site, including the Internet Archive. There is nothing you can add to your robots.txt file that will increase your site traffic, only getting more offsite links to your site or including more content with keywords will do that.

For more on Webmaster Tools, see this FAQ:

https://support.google.com/webmasters/?hl=en#topic=3309469