Sorry about the delay between my last post and this, it’s been a busy couple of weeks.

Today I just wanted to outline a mistake (or missed opportunity) which is currently visible in my clients, and others, websites.

Many of you will already know about XML sitemaps, and their uses (for example aiding search engines to deep root your website) and I’m sure the majority of you currently have one in place and have added them on Google Webmaster Tools.

My problem is the amount of web developers who forget to add the link to their sitemap onto their robots.txt file. There is more then one search engine out there, and by only adding the link to Google Webmaster Tools you are only helping your website on Google and not the countless other search engines.

By simply including this line onto your robots file you are informing all crawlers where to find your XML sitemap:

Sitemap: http://www.yourdomain.com/sitemap.xml

It is important to make sure this line is correctly formatted as above, including the full link to the file (which could be sitemap.xml).

By doing this not only are you saving yourself the job of adding the xml sitemap to Google directly, but you are also exposing your sitemap to Ask, Bing, Yahoo .etc.

Anyway, on the same note there are plenty of other things you can do to encourage search engines to visit your sitemap xml file, I will post more on this next time, including a script which will automatically “ping” selected search engines for you.

Until next time

Dave.