Spiders, Robots, Oh My!

Search engines find your web site using little programs called "robots" or "spiders" that search the internet for new content. These are busy little guys, continually scouring the internet for the latest and greatest information. If you know how to cater to them you'll get your site noticed by search engines.

First and foremost there's one concept that you cannot go wrong with:

The more human-friendly your site is, the more search engine friendly it will be.

It's sort of a simple----search engine spiders are trying to look at each web site they encounter as if they were human. If something is good or helpful for a human visitor, a search engine will give you a better rank. Here's what I mean:

Humans like fresh content. Human visitors like new content to read. You wouldn’t like reading the same article every month in your favorite magazine, would you? If you put new pages and text on your web site on a regular basis, people will come to your site more often to read it.

Likewise, the more often you update and add new content the more often search engines will search your site and index your new content. If you post new stories on your blog a couple times a week, Google might visit your site once a week to see what is new and add it to their search results. Google will visit larger sites (like, for example, CNN.com) several times a day because the web site is updated continually throughout the day.

So, to get your web site visited regularly by a spider, you want to update your site as often as possible. If you stop adding new content for 6 six months, a spider will act like a human visitor would---they'll visit less and less frequently until they stop visiting completely.

Making content accessible:

Humans like to be able to find new content. If you create a new page and you put a link from your home page to that new page, it's easy for people to find the new page. Whereas if you put a new page on your site but you don't link to it, human visitors can't find that page.

Spiders find content by following links too. Once you've got a new page on your site, you want to make sure there is a link to the new content so a spider can find it.

One thing you can do to make sure search engines visit all the important pages on your site is create a site map. A site map is simply a list of links to each of the important pages on your site.

A search engine that finds this link will quickly gain access to all the pages on the web site. The spider can just go down the list. A site map is also helpful for your human visitors. If someone cannot find the information they are looking for, a quick visit to the site map can point them in the right direction.

Outgoing links:

If you're giving a human visitor useful information, you would probably link to other resources around the internet that would also be useful for your visitors to visit. If you have a lot of content, it's expected that it will include links that leave your site. Without outgoing links you become a stagnant dead in the water website---links go in but they don't go out. Since links to other resources off of your site are helpful to humans, search engine spiders want to see them on your site.

For more information on this, you may want to visit these links:

http://en.wikipedia.org/wiki/Sitemaps

http://webthang.org/downloads/RobotsReferenceWorksheet.pdf

http://www.barkerenterprisesinc.com/uploads/Spiders_-_CEM_08.09.pdf

http://www.epdfsearch.com/en/pdf/SEO+Tips-3.html