How to use meta robot tags

In a site audit of a small media site recently, I found that of the 500 content pages that it has, Google had only indexed five or so pages.

This seemed strange, so upon checking the code, we found the following embedded near the top of every page: <meta name=”robots” content=”noindex, nofollow”/>

This is called the meta robots tag, and the ‘noindex, nofollow’ command basically tells Google not to include the site in Google’s index and not to follow any links on the page.

Because this particular site had the meta robots tag on the top of every page on the site, none of it was getting any search traffic as Google had ignored the entire site.

So what is the meta robots tag used for? In some cases, there are pages that you wouldn’t want Google to include in their index, such as a privacy notice or a terms and conditions page. Using this tag will omit those pages from Google.

What you need to do is a quick site audit of your own site to make sure that your content is being indexed properly.

You can first check what Google thinks your site looks like and then view the source code to make sure that your meta robots tag is not misconfigured.

You can typically do this by right clicking on a blank spot on the page, click on “View Page Source” and look out for the above meta robots tag.