Sat Jan 05 15:02:47 PST 2019

280
All search spiders are not the same though, and some other things can keep your
site from getting indexed. Yahoo!, for example, has been known not to index
some sites well if each page does not have a unique title.
Make sure each page of your site starts with unique, page-relevant content, and is
linked to from other pages on your site.
One URL for Each Unique Content Unit
At the end of the day, search engines want as much useful valuable content as they
can index. Each additional useful document improves their relevancy, market-
share, and allows them to sell more ad space.
Some people use rotating URLs or create many pages using printer-friendly
versions and different versions based on size and color. If many of your pages are
almost exactly the same, or if the URLs that search engines index are usually dead
when they go back to re-index them, there stands a good chance that search
engines will not want to index your site very well.
Limit the options you offer search engines to the pages you actually want indexed.
Disable Cookies
Search engines do not accept cookies. If your site gives cookies, try turning
cookies off and see your site how some search engines see it. If, when cookies are
disabled, a session ID is placed in the URL to search spiders, it could cause
indexing problems.
Dynamic Site Tips
To get dynamic content spidered, I recommend doing the following:
? Use two or less variables if possible. If you use many variables, you
may also want to try rewriting your URLs.
? Keep each product or category ID variable at or below ten digits.
? The variable id= is often associated with session IDs. Search
engines do not want to index multiple copies of the exact same content
and may avoid portions of your site if they think that a parameter is a
session ID. Avoid using ID in your URL. Instead of using ID=345 or
catID=345, use prod=345 or cat-345 in the URL.
? If you are having indexing issues, your site is new and is not yet
indexed, or you are doing a major infrastructure update, you may want
to rewrite your URLs using descriptive clean static looking URLs over
ugly variable filled URLs.
? Building a site map and linking to related pages can also help spiders
find your inner pages.