What Is a Meta Robots Tag?
A meta robots tag is a bit of HTML code that tells search engine robots tips on how to crawl, index, and show a web page’s content material.
It goes within the <head> part of the web page and might appear to be this:
<meta title="robots" content material="noindex">
The meta robots tag within the instance above tells all search engine crawlers to not index the web page.
Let’s talk about what you should use robots meta tags for, why they’re essential for search engine optimization, and tips on how to use them correctly.
Meta Robots vs. Robots.txt
Meta robots tags and robots.txt information have related features however serve totally different functions.
A robots.txt file is a single textual content file that applies to all the web site. And tells search engines like google which pages to crawl.
A meta robotstag applies to solely the web page containing the tag. And tells search engines like google tips on how to crawl, index, and show info from that web page solely.
What Are Robots Meta Tags Used For?
Robots meta tags assist management how Google crawls and indexes a web page’s content material. Together with whether or not to:
- Embrace a web page in search outcomes
- Comply with the hyperlinks on a web page
- Index the photographs on a web page
- Present cached outcomes of the web page on the search engine outcomes pages (SERPs)
- Present a snippet of the web page on the SERPs
Beneath, we’ll discover the attributes you should use to inform search engines like google tips on how to work together together with your pages.
However first, let’s talk about why robots meta tags are essential and the way they will have an effect on your web site’s search engine optimization.
How Do Robots Meta Tags Have an effect on search engine optimization?
Robots meta tags assist Google and different search engines like google crawl and index your pages effectively.
Particularly for giant or regularly up to date websites.
In spite of everything, you possible don’t want each web page in your web site to rank.
For instance, you in all probability don’t need search engines like google to index:
- Pages out of your staging web site
- Affirmation pages, reminiscent of thanks pages
- Admin or login pages
- Inner search consequence pages
- Pages with duplicate content material
Combining robots meta tags with different directives and information, reminiscent of sitemaps and robots.txt, can subsequently be a helpful a part of your technical search engine optimization technique. As they can assist stop points that would in any other case maintain again your web site’s efficiency.
What Are the Title and Content material Specs for Meta Robots Tags?
Meta robots tags include two attributes: title and content material. Each are required.
Title Attribute
This attribute signifies which crawler ought to observe the directions within the tag.
Like this:
title="crawler"
If you wish to deal with all crawlers, insert “robots” because the “title” attribute.
Like this:
title="robots"
Word
The title attribute isn’t case-sensitive. So “robots,” “ROBOTS,” and “Robots” will all work.
If you wish to prohibit crawling to particular search engines like google, the title attribute helps you to do this. And you’ll select as many (or as few) as you need.
Listed below are a number of widespread crawlers:
- Google: Googlebot (or Googlebot-news for information outcomes)
- Bing: Bingbot (see the checklist of all Bing crawlers)
- DuckDuckGo: DuckDuckBot
- Baidu: Baiduspider
- Yandex: YandexBot
Word
Whereas main search engines like google will obey your meta robots tags, there’s a probability that others won’t. This implies you shouldn’t use meta robots tags as a safety measure on delicate content material. And as a substitute go for a safer methodology like password safety.
Content material Attribute
The “content material” attribute incorporates directions for the crawler.
It appears like this:
content material="instruction"
Word
Just like the title, the content material attribute additionally isn’t case-sensitive.
Google helps the next “content material” values:
Default Content material Values
And not using a robots meta tag, crawlers will index content material and observe hyperlinks by default (except the hyperlink itself has a “nofollow” tag).
This is identical as including the next “all” worth (though there isn’t any must specify it):
<meta title="robots" content material="all"
So, in the event you don’t need the web page to look in search outcomes or for search engines like google to crawl its hyperlinks, you might want to add a meta robots tag. With correct content material values.
Noindex
The meta robots “noindex” worth tells crawlers to not embrace the web page within the search engine’s index or show it within the SERPs.
<meta title="robots" content material="noindex">
With out the noindex worth, search engines like google might index and serve the web page within the search outcomes.
Typical use instances for “noindex” are cart or checkout pages on an ecommerce web site.
Nofollow
This tells crawlers to not crawl the hyperlinks on the web page.
<meta title="robots" content material="nofollow">
Google and different search engines like google usually use hyperlinks on pages to find these linked pages. And hyperlinks can assist move authority from one web page to a different.
Use the nofollow rule in the event you don’t need the crawler to observe any hyperlinks on the web page or move any authority to them.
This could be the case in the event you don’t have management over the hyperlinks positioned in your web site. Equivalent to in an unmoderated discussion board with largely user-generated content material.
Word
This doesn’t stop Google from ever discovering the linked pages, as they could be linked to from different pages and web sites.
Noarchive
The “noarchive” content material worth tells Google to not serve a duplicate of your web page within the search outcomes.
<meta title="robots" content material="noarchive">
When you don’t specify this worth, Google might present a cached copy of your web page that searchers might even see within the SERPs.
You possibly can use this worth for time-sensitive content material, inside paperwork, PPC touchdown pages, or some other web page you don’t need Google to cache.
Noimageindex
This worth instructs Google to not index the photographs on the web page.
<meta title="robots" content material="noimageindex">
Utilizing “noimageindex” might damage potential natural visitors from picture outcomes. And if customers can nonetheless entry the web page, they’ll nonetheless be capable of discover the photographs. Even with this tag in place.
Notranslate
“Notranslate” prevents Google from serving translations of the web page in search outcomes.
<meta title="robots" content material="notranslate">
When you don’t specify this worth, Google can present a translation of the title and snippet of a search consequence for pages that aren’t in the identical language because the search question.
If the searcher clicks the translated hyperlink, all additional interplay is thru Google Translate. Which robotically interprets any adopted hyperlinks.
Use this worth in the event you choose to not have your web page translated by Google Translate.
For instance, if in case you have a product web page with product names you don’t need translated. Or in the event you discover Google’s translations aren’t all the time correct.
Nositelinkssearchbox
This worth tells Google to not generate a search field in your web site in search outcomes.
<meta title="robots" content material="nositelinkssearchbox">
When you don’t use this worth, Google can present a search field in your web site within the SERPs.
Like this:
Use this worth in the event you don’t need the search field to look.
Nosnippet
“Nosnippet” stops Google from displaying a textual content snippet or video preview of the web page in search outcomes.
<meta title="robots" content material="nosnippet">
With out this worth, Google can produce snippets of textual content or video based mostly on the web page’s content material.
The worth “nosnippet” additionally prevents Google from utilizing your content material as a “direct enter” for AI Overviews. However it’ll additionally stop meta descriptions, wealthy snippets, and video previews. So use it with warning.
Whereas not a meta robots tag, you should use the “data-nosnippet” attribute to stop particular sections of your pages from displaying in search outcomes.
Like this:
<p>This textual content might be proven in a snippet
<span data-nosnippet>however this half would not be proven</span>.</p>
Max-snippet
“Max-snippet” tells Google the utmost character size it could actually present as a textual content snippet for the web page in search outcomes.
This attribute has two essential instances to concentrate on:
- 0: Opts your web page out of textual content snippets (as with “nosnippet”)
- -1: Signifies there’s no restrict
For instance, to stop Google from displaying a textual content snippet within the SERPs, you could possibly use:
<meta title="robots" content material="max-snippet:0">
Or, if you wish to permit as much as 100 characters:
<meta title="robots" content material="max-snippet:100">
To point there’s no character restrict:
<meta title="robots" content material="max-snippet:-1">
Max-image-preview
This tells Google the utmost measurement of a preview picture for the web page within the SERPs.
There are three values for this directive:
- None: Google received’t present a preview picture
- Customary: Google might present a default preview
- Giant: Google might present a bigger preview picture
<meta title="robots" content material="max-image-preview:giant">
Max-video-preview
This worth tells Google the utmost size you need it to make use of for a video snippet within the SERPs (in seconds).
As with “max-snippet,” there are two essential values for this directive:
- 0: Opts your web page out of video snippets
- -1: Signifies there’s no restrict
For instance, the tag under permits Google to serve a video preview of as much as 10 seconds:
<meta title="robots" content material="max-video-preview:10">
Use this rule if you wish to restrict your snippet to indicate sure elements of your movies. When you don’t, Google might present a video snippet of any size.
Indexifembedded
When used together with noindex, this (pretty new) tag lets Google index the web page’s content material if it’s embedded in one other web page by way of HTML components reminiscent of iframes.
(It wouldn’t have an impact with out the noindex tag.)
<meta title="robots" content material="noindex, indexifembedded">
“Indexifembedded” has been created with media publishers in thoughts:
They usually have media pages that shouldn’t be listed. However they do need the media listed when it’s embedded in one other web page’s content material.
Beforehand, they might have used “noindex” on the media web page. Which might stop it from being listed on the embedding pages too. “Indexifembedded” solves this.
Word
Not all search engines like google help this tag.
Unavailable_after
The “unavailable_after” worth prevents Google from displaying a web page within the SERPs after a particular date and time.
<meta title="robots" content material="unavailable_after: 2024-10-21">
You could specify the date and time utilizing RFC 822, RFC 850, or ISO 8601 codecs. Google ignores this rule in the event you don’t specify a date/time. By default, there isn’t any expiration date for content material.
You should use this worth for limited-time occasion pages, time-sensitive pages, or pages you now not deem essential. This features like a timed noindex tag, so use it with warning. Or you could possibly find yourself with indexing points later down the road.
Combining Robots Meta Tag Guidelines
There are two methods in which you’ll mix robots meta tag guidelines:
- Writing a number of comma-separated values into the “content material” attribute
- Offering two or extra robots meta components
A number of Values Contained in the ‘Content material’ Attribute
You may combine and match the “content material” values we’ve simply outlined. Simply be sure to separate them by comma. As soon as once more, the values usually are not case-sensitive.
For instance:
<meta title="robots" content material="noindex, nofollow">
This tells search engines like google to not index the web page or crawl any of the hyperlinks on the web page.
You may mix noindex and nofollow utilizing the “none” worth:
<meta title="robots" content material="none">
However some search engines like google, like Bing, don’t help this worth.
Word
When you mix conflicting directives or if one is a subset of the opposite (like “nosnippet, max-snippet: -1”), Google will use whichever is most restrictive. On this instance, the nosnippet rule would apply.
Two or Extra Robots Meta Parts
Use separate robots meta components if you wish to instruct totally different crawlers to behave in a different way.
For instance:
<meta title="robots" content material="nofollow"><meta title="YandexBot" content material="noindex">
This mix instructs all crawlers to keep away from crawling hyperlinks on the web page. However it additionally tells Yandex particularly to not index the web page (along with not crawling the hyperlinks).
Search Engine Assist for Meta Robots Tags
The desk under reveals the supported meta robots values for various search engines like google:
Worth
Bing
Yandex
noindex
Y
Y
Y
noimageindex
Y
N
N
nofollow
Y
N
Y
noarchive
Y
Y
Y
nocache
N
Y
N
nosnippet
Y
Y
N
nositelinkssearchbox
Y
N
N
notranslate
Y
N
N
max-snippet
Y
Y
N
max-video-preview
Y
Y
N
max-image-preview
Y
Y
N
indexifembedded
Y
N
N
unavailable_after
Y
N
N
Find out how to Implement Robots Meta Tags
Including Robots Meta Tags to Your HTML Code
When you can edit your web page’s HTML code, add your robots meta tags into the <head> part of the web page.
For instance, in order for you search engines like google to keep away from indexing the web page and to keep away from crawling hyperlinks, use:
<meta title="robots" content material="noindex, nofollow">
Implementing Robots Meta Tags in WordPress
When you’re utilizing a WordPress plugin like Yoast search engine optimization, open the “Superior” tab within the block under the web page editor.
Set the “noindex” directive by switching the “Permit search engines like google to indicate this web page in search outcomes?” drop-down to “No.”
Or stop search engines like google from following hyperlinks by switching the “Ought to search engines like google observe hyperlinks on this web page?” to “No.”
For different directives, you must implement them within the “Meta robots superior” discipline.
Like this:
When you’re utilizing Rank Math, choose the robots directives straight from the “Superior” tab of the meta field.
Like so:
Including Robots Meta Tags in Shopify
To implement robots meta tags in Shopify, edit the <head> part of your theme.liquid structure file.
To set the directives for a particular web page, add the code under to the file:
{% if deal with incorporates 'page-name' %}
<meta title="robots" content material="noindex">
{% endif %}
This instance instructs search engines like google to not index /page-name/ (however to nonetheless observe all of the hyperlinks on the web page).
You could create separate entries to set the directives throughout totally different pages.
Word
Be extraordinarily cautious when enhancing theme information. Errors right here can considerably hurt your web site. When you’re uncomfortable with this threat, ask your developer for assist.
Implementing Robots Meta Tags in Wix
Open your Wix dashboard and click on “Edit Website.”
Click on “Pages & Menu” within the left-hand navigation.
Within the tab that opens, click on “…” subsequent to the web page you wish to set robots meta tags for. Select “search engine optimization fundamentals.”
Then click on “Superior search engine optimization” and click on on the collapsed merchandise “Robots meta tag.”
Now you possibly can set the related robots meta tags in your web page by clicking the checkboxes.
When you want “notranslate,” “nositelinkssearchbox,” “indexifembedded,” or “unavailable_after,” click on “Extra tags”and “Add New Tags.”
Now you possibly can paste your meta tag in HTML format.
What Is the X-Robots-Tag?
An x-robots-tag serves the identical perform as a meta robots tag however for non-HTML information. Equivalent to photographs and PDFs.
You embrace it as a part of the HTTP header response for a URL.
Like this:
To implement the x-robots-tag, you will must entry your web site’s header.php, .htaccess, or server configuration file. You should use the identical guidelines as these we mentioned earlier for meta robots tags.
Find out how to Implement X-Robots-Tags
Utilizing X-Robots-Tag on an Apache Server
To make use of the x-robots-tag on an Apache net server, add the next to your web site’s .htaccess file or httpd.conf file.
<Recordsdata ~ ".pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Recordsdata>
For instance, the code above instructs search engines like google to not index or to observe any hyperlinks on all PDFs throughout all the web site.
Utilizing X-Robots-Tag on an Nginx Server
When you’re working an Nginx server, add the code under to your web site’s .conf file:
location ~* .pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
The instance code above will apply noindex and nofollow values to all the web site’s PDFs.
Frequent Meta Robots Tag Errors to Keep away from
Let’s check out some widespread errors to keep away from when utilizing meta robots and x-robots-tags:
Utilizing Meta Robots Directives on a Web page Blocked by Robots.txt
When you disallow crawling of a web page in your robots.txt file, main search engine bots received’t crawl it. So any meta robots tags or x-robots-tags on that web page might be ignored.
Guarantee search engines like google can crawl any pages with meta robots tags or x-robots-tags.
Including Robots Directives to the Robots.txt File
Though by no means formally supported by Google, you have been as soon as in a position so as to add a “noindex” directive to your web site’s robots.txt file.
That is now not an choice, as confirmed by Google.
The “noindex” rule in robots meta tags is the simplest strategy to take away URLs from the index whenever you do permit crawling.
Eradicating Pages with a Noindex Directive from Sitemaps
When you’re attempting to take away a web page from the index utilizing a “noindex” directive, depart the web page in your sitemap till it has been eliminated.
Eradicating the web page earlier than it’s deindexed may cause delays in deindexing.
Not Eradicating the ‘Noindex’ Directive from a Staging Atmosphere
Stopping robots from crawling pages in your staging web site is a greatest follow. However it’s simple to overlook to take away “noindex” as soon as the location strikes into manufacturing.
And the outcomes may be disastrous. As search engines like google might by no means crawl and index your web site.
To keep away from these points, verify that your robots meta tags are appropriate earlier than shifting your web site from a staging platform to a dwell atmosphere.
Find out how to Test Your Web site for Meta Robots Tag Points
Discovering and fixing crawlability points (and different technical search engine optimization errors) in your web site can dramatically enhance efficiency.
When you don’t know the place to start out, use Semrush’s Website Audit device.
Simply enter your area and click on “Begin Audit.”
You may configure varied settings, just like the variety of pages to crawl and which crawler you’d like to make use of. However it’s also possible to simply depart them as their defaults.
While you’re prepared, click on “Begin Website Audit.”
When the audit is full, head to the “Points” tab.
Within the search field, sort “blocked from crawling” to see errors concerning your meta robots tags or x-robots-tags.
Like this:
Click on on “Why and tips on how to repair it” subsequent to a difficulty to learn extra concerning the concern and tips on how to repair it.
Repair every of those points to enhance your web site’s crawlability. And to make it simpler for Google to search out and index your content material.
FAQs
When Ought to You Use the Robots Meta Tag vs. X-Robots-Tag?
Use the robots meta tag for HTML pages and the x-robots-tag for different non-HTML sources. Like PDFs and pictures.
This isn’t a technical requirement. You possibly can inform crawlers what to do together with your webpages by way of x-robots-tags. However it’s simpler to attain the identical factor by implementing the robots meta tags on a webpage.
You too can use x-robots-tags to use directives in bulk. Fairly than merely on a web page stage.
Do You Must Use Each Meta Robots Tag and X-Robots-Tag?
You don’t want to make use of each meta robots tags and x-robots-tags. Telling crawlers tips on how to index your web page utilizing both a meta robots or x-robots-tag is sufficient.
Repeating the instruction received’t enhance the possibilities that Googlebot or some other crawlers will observe it.
What Is the Best Approach to Implement Robots Meta Tags?
Utilizing a plugin is normally the simplest approach so as to add robots meta tags to your webpages. As a result of it doesn’t normally require you to edit any of your web site’s code.
Which plugin it is best to use depends upon the content material administration system (CMS) you’re utilizing.
Use Meta Robots Tags Accurately to Keep away from Indexing Points
Robots meta tags ensure that the content material you’re placing a lot effort into will get listed. If search engines like google don’t index your content material, you possibly can’t generate any natural visitors.
So, getting the fundamental robots meta tag parameters proper (like noindex and nofollow) is totally essential.
Test that you just’re implementing these tags accurately utilizing Semrush Website Audit.
This publish was up to date in 2024. Excerpts from the unique article by Carlos Silva might stay.