Off Page SEO: Technical Pages

This post is a follow-up of last week post about SEO: On Page SEO: Search Engine Unfriendly Technologies. I invite you to read this previous post as well. If you would like to have a whole picture of my SEO guide, you can check the series from the very first post: Comprehensive introduction to SEO.

Robots.txt File

The Robots.txt file is used only by Search Engines. It is located in the root folder of your website. It has 2 goals: Give the Sitemap Url to the Spiders, tell the Search Engine Robots which part of your website they shouldn’t index.

Here is an example Robots.txt:

1
2
3
4
5
6
7
8
9

User-Agent: *
Disallow: /cgi/
Disallow: /admin/
Disallow: /content/this-article.html
Sitemap: http://www.mysite.com/sitemap.xml

Sitemap XML

Sitemap is an internet protocol which is widely accepted by Search Engines. The aim of the Sitemap is to feed the Search Engine Robots in an automated way the Urls of the new pages to index.

This improves drastically the speed of indexation of your newest pages, which is very useful when you launch a new website.

You can find all the useful information about Sitemaps on http://sitemaps.org which also provides a file validator to check the coding of your sitemap file.

You can generate this file manually, but it is better to have it generated dynamically when your website is updated. It is recommended to put the freshest urls at the top of your Sitemap file.

A Sitemap shouldn’t include more than 50 000 Urls and weight more than 50Mo. In the case you have more Urls, you should have additional Sitemap files regrouped in a Sitemap Index.

The Sitemap.xml file should be declared in the Robots.txt (located in the root folder).

Added to the classical Sitemap, you can add specific Sitemaps for some types of content.

Google News Sitemap

It will highly help regarding the speed of indexation of your articles in Google News. It is limited to 1000 urls and it must include only “news-type” articles. Also, it should be updated in real time, every time you post a new article.

Video Sitemap

It will let you submit your videos while associating them with meta-data and a snapshot which will be shown in the SERP.

Geolocalised Sitemap

It is indexing the files with geographic information like KLM and GeoRSS which will be shown in apps like Google Maps and Google Earth.