Robots Meta Tag

What is a robots meta tag?

A robots meta tag provides indexing and serving rules for individual pages to search engine bots.

Robots no index — A robots meta tag directing search engine bots to not index the page

Why robots meta tags matter for SEO

Unlike robots exclusion, which requests bots not to crawl pages, robots meta tags provide page-by-page granular indexing and serving rules when a page is crawled. The rules can specify if the page should be indexed, if the links should be followed, and if the page should be cached. The rules can also influence search result appearance, translation, and embedded iframe content.

Robots meta tag rules used by Google

all – No restrictions
noindex – Don’t index
nofollow – Don’t follow links
none – Don’t index or follow links
noarchive – Don’t cache in search results
nositelinkssearchbox – Don’t show a sitelinks search box
nosnippet – Don’t show a text snippet or video preview
indexifembedded – Allows indexing of embeds on pages when the embed source also uses noindex
max-snippet: [number] – Limits or excludes text snippets in search results
max-image-preview: [number] – Specifies the size of the image preview or requests no image preview
max-video-preview: [number] – Specifies the number of seconds for a video preview or requests no video preview
notranslate – Don’t translate page title and meta description
noimageindex – Don’t index images on page
unavailable_after [date/time] – Specifies data and time to begin excluding page from search results

Robots meta tag tips

The unintentional use of the noindex directive on pages is a common cause for pages not being indexed, especially on enterprise sites. Always check the page to ensure it’s not being used.
A robots meta tag won’t keep a page from being crawled. If you don’t want a page to be crawled, exclude it in robots.txt or password-protect it.