Configure the Robot Tags

Configuring Robot Tags in Magento 2

Robot meta tags are essential for controlling how search engines crawl and index your Magento 2 store. This configuration screen allows you to define default robot directives for category pages, product pages, and CMS pages, giving you precise control over your SEO strategy.

Understanding Robot Tags

Robot tags instruct search engines on whether to index and follow links on specific pages. Common robot directives include:

• INDEX, FOLLOW: Allows indexing of the page and following the links.
• NOINDEX, FOLLOW: Prevents the page from being indexed but allows search engines to follow links on the page.
• INDEX, NOFOLLOW: The page is indexed, but links are not followed.
• NOINDEX, NOFOLLOW: Neither the page is indexed nor its links followed.

How to Configure Robot Tags

General Settings

• Version: Displays the installed plugin version.
• Enabled [Store View]: Choose whether to enable or disable the plugin. If set to No, default Magento robot tag configurations will apply.

Settings Section

This section lets you assign default robot tags to various page types:

• Category Pages:
  • Default robots for category pages.
  • Recommended: INDEX, FOLLOW if you want categories indexed, but set NOINDEX for categories with duplicate or low-value content.
• Product Pages:
  • Default robots for individual product pages.
  • Recommended: INDEX, FOLLOW for pages with high-ranking products.
• CMS Pages:
  • Default robots for content pages like blogs, static pages, etc.
  • Adjust based on content importance. Use NOINDEX for temporary or irrelevant pages.
• Use System Value: Check this box to apply the default system value or uncheck to customize the robots setting.

Logging and Debugging

• Logging: Enable logging to track debug events, API calls, and errors in the debug.log file.
• Debug Mode [Store View]: Enable this for troubleshooting, but disable it in production environments for performance.

Best Practices for Robot Tags in Magento 2

• Use NOINDEX for duplicate or low-priority pages to optimize crawl budgets.
• Apply INDEX, FOLLOW to essential pages like product listings, categories, and high-conversion landing pages.
• Monitor pages using tools like Google Search Console to ensure the correct pages are indexed.