SEO Secrets | Web Spiders/Crawlers & Robots.txt Files
Spiders, Crawlers, and Robots on the Internet?
What are Website Spiders?
Website crawling by website spiders is comparable to what a spider is real life does. Google, and search engines like it, index your website by using these website crawlers which periodically come to your site.
How do they relate to websites?
Search engines use these website spiders to find your website and index it for future searches by search engine users. There a couple different ways a search engine may find your website and its pages to index. One key to allow your site to be found is by having fresh new content and/or content rich pages. Search engines don’t want to index old and irrelevant information. A second key to get your website on Google is providing related article links, and good ones, on your page. Providing good links to sites with a good reputation is very important, because those sites will provide you with positive linking to your own site.
Google, for instance, begins the website indexing process by using previous lists of URLs which had been generated from previous crawls. Starting from these previously known sites, the crawlers continue on links from these sites. Links which are shorter in length are the ones most likely to be followed, as crawlers deem those to be the most important. As you can see, links are the basis in web crawling and therefore a key for directing web spiders.
General Process of Web Crawling:
Search engine spiders come to your site:
- via links from other sites
- your site’s previous relevance in the web directory of the search engine
- due to the freshness of your site
Once on your website, the website spiders will read the content of each page. The website crawlers site meta tags as well, and then follow the links that the internet site connects with if allowed. The website spiders then return back to the depository, where the information is indexed. Those links which are on your page will be indexed as well, unless meta tags prevent them. This process is discussed in the Follow/No Follow blog by Shazzam-Media. Website spiders return periodically to check for any changes in information. Regulatory of this depends on the moderators of the search engine, and it is something you can’t control. Crawl frequency and depth are things which you can control.
Submitting a sitemap to the search engine is a major influence on the engines ability to better crawl your site. The sitemap allows the engine to understand your website once the web spiders come to your website, and helps provide changes on the site to be indexed quicker.
What is a Robot.txt (robot text files)?
Robot.txt files are a website tool which can tell the spiders which pages to index. By placing a robot.txt on the website server with either allow or disallow tag, it provides a set of instructions to the web crawlers which visit the page. Essentially, robots.txt files are used to map out your site to the crawlers for what they can and cannot index.
The benefits of using a robot text file: 
- Avoids wastage of server resources
- Prevent website crawler access to pages which you don’t want to be indexed such as login pages
- Prevent search engines from indexing pages which are still being built
One item to consider is if you want your entire website to be readily available to website spiders placing the robot.txt in your server isn’t necessary; not even one which allows everything. Leaving the file out completely is the best decision to go with.
By knowing your website and providing search engines with quality pages will allow your site to get to the top of Google, and by properly using robot.txt files will help to improve website navigation.
Shazzam-Media is a company specializing in search engine optimization which offers the best in website design.
Related Articles:
http://www.stateofsearch.com/how-to-influence-search-robots-in-crawling-your-website/
http://dhall-online.com/how-do-engines-like-google-work-web-spiders/ \
http://www.robotstxt.org/faq.html
http://www.seoconsultants.com/robots-text-file/
http://www.thesitewizard.com/archive/robotstxt.shtml
http://www.google.com/support/webmasters/bin/answer.py?answer=156449