Wed Dec 26 15:03:30 PST 2018

122
Googlebot also supports wildcards in the robots.txt. The following would stop
Googlebot from reading any URL that includes the string ?sort=? no matter where
that string occurs in the URL:
User-agent: Googlebot
Disallow: /*sort=
In 2006 Yahoo! also added robots.txt wildcard support. Their example pages is
useful for helping you understand how to structure your robots.txt file
http://www.ysearchblog.com/archives/000372.html
You have to be careful when changing your robots.txt file, because the
following code
Disallow: /*page
also blocks a file like this from being indexed in Google
beauty-pageants.php
Google?s Webmaster Toolset shows you what pages they have tried crawling that
you have already blocked via robots.txt, and they have a robots.txt testing tool
which will show you if a specific URL would end up getting blocked by your
robots.txt file.
In 2007 Google released an unavailable_after meta tag, which tells Google to not
crawl a URL after a specific date. I do not recommend using this. Instead, if one of
your old URLs ranks where you would like a new one to rank I recommend 301
redirecting it to the URL you want to rank.
NOODP & NOYDIR
If your site is listed in the Yahoo Directory then Yahoo may display information
from the directory listing rather than displaying your page title and meta
description.
If your site is listed in the Open Directory Project then Google may display
information from the directory listing rather than displaying your page title and
meta description.
The problem with displaying this third party data is that it is much less likely to be
brand friendly and sales oriented. If your site is listed in these directories you can
still tell the search engines to display your information by using the following tags:


Free Meta Tag Generator
I offer a free page title meta tag generator here. I wrote general tips on that tools
page to help you create your title and meta tags. Find it here:
http://tools.seobook.com/meta-medic/