Blog: May 2010

Remove URLs using a robots.txt

Robots.txt and x-robots

It's likely that you have pages on your website that you don't want to be indexed by the search engines. These could be pages like your privacy policy or simply pages that you don't want to be accessible to the public. In the first case if the said page is a page like the privacy policy which is linked and accessible via your website then the page can be blocked using a robots.txt file.

More on Remove URLs using a robots.txt

SEO URL Canonicalization

URL canonicalization

Website URL canonicalization is an important part of Onpage SEO Optimisation. When a website is indexed by a search engine they need to choose the best URL to index for that domain. A website can normally be accessed from a number of different URLs which all point to the same domain for example:

More on SEO URL Canonicalization