Google Sitemap & Robots

Version v1.2.0
Platform Magento 2
Released 2 August 2025

Robots Configuration for SEO

Complete settings reference for the Robots Meta Tag functionality in the Google Sitemap & Robots extension. This guide covers how to control search engine indexing behavior for your pages.

Overview

Navigate to Magmodules → Robots to access robots settings.

The robots module lets you set default meta robots tags for different page types. This controls whether search engines index pages and follow links on them.

General

Setting Description
Enabled Enable or disable the robots module

Understanding Robots Tags

Available Options

Tag Meaning
INDEX,FOLLOW Index this page, follow all links
NOINDEX,FOLLOW Don't index this page, but follow links
INDEX,NOFOLLOW Index this page, don't follow links
NOINDEX,NOFOLLOW Don't index, don't follow links

When to Use Each

INDEX,FOLLOW (default for most pages)

  • Product pages you want in search results
  • Category pages with unique content
  • Important CMS pages (about us, contact)

NOINDEX,FOLLOW

  • Filtered category pages (price, color, size filters)
  • Search results pages
  • Paginated pages (page 2, 3, etc.)
  • Thank you / confirmation pages

INDEX,NOFOLLOW

  • Pages with user-generated links you don't trust
  • Rarely needed in e-commerce

NOINDEX,NOFOLLOW

  • Admin or internal pages
  • Duplicate content pages
  • Staging/test pages

Settings

Set default robots meta tags for different page types:

Category Pages

Setting Description
Category Pages - Default Robots Default robots tag for all category pages

Recommended: INDEX,FOLLOW

Categories are important for SEO. Only use NOINDEX for filtered views or thin content categories.

Product Pages

Setting Description
Product Pages - Default Robots Default robots tag for all product pages

Recommended: INDEX,FOLLOW

Product pages should be indexed. Use product-level overrides for items you want to exclude.

CMS Pages

Setting Description
CMS Pages - Default Robots Default robots tag for all CMS pages

Recommended: INDEX,FOLLOW

Most CMS pages should be indexed. Override individually for pages like "Terms & Conditions" if needed.

Debug

Setting Description
Debug Mode Enable detailed logging for robots settings

Enable when troubleshooting robots tag issues. Check var/log/ for output.

Page-Level Overrides

The extension sets defaults, but you can override per page:

Categories

  1. Edit category in Catalog → Categories
  2. Go to Search Engine Optimization section
  3. Set Robots field to override default

Products

  1. Edit product in Catalog → Products
  2. Go to Search Engine Optimization section
  3. Set Robots field to override default

CMS Pages

  1. Edit page in Content → Pages
  2. Go to Search Engine Optimization section
  3. Set Robots field to override default

Common Scenarios

Scenario 1: Hide Filtered Category Pages

You want main category pages indexed but not filtered views.

Configuration:

  • Default Category Robots: INDEX,FOLLOW
  • Use layered navigation that adds NOINDEX to filtered URLs

Scenario 2: Exclude Out-of-Stock Products

You don't want out-of-stock products in search results.

Approach:

  • Keep default as INDEX,FOLLOW
  • Use product attribute or automation to set NOINDEX on out-of-stock items

Scenario 3: New Store Launch

You're preparing a store but don't want it indexed yet.

Configuration:

  • Set all defaults to NOINDEX,NOFOLLOW
  • Change to INDEX,FOLLOW when ready to launch

Integration with Sitemap

The robots module works together with the sitemap module:

  • Pages marked NOINDEX can be excluded from sitemap automatically
  • Enable "Exclude No-Index" options in sitemap settings
  • This prevents sending mixed signals to search engines

Verifying Robots Tags

Check in Browser

  1. Open your page
  2. View page source (Ctrl+U or Cmd+U)
  3. Look for: <meta name="robots" content="INDEX,FOLLOW">

Using Google Search Console

  1. Go to URL Inspection
  2. Enter your page URL
  3. Check "Indexing allowed?" status

Need More Help?

Documentation:

Support:

Created:
Updated:
star star star star star
star star star star star
Alexandru-Manuel Carabus
Magmodules sets the bar for Magento module quality and support—we check their catalog first for client feature requests, and they’re our first choice for licenses.
Google 11 Nov 2025
star star star star star
star star star star star
Matt Austin
Possibly the fastest support response times of any Magento Extension vendor. Great extensions too!
Google 09 Sep 2025
star star star star star
star star star star star
Jan Privé
Dankzij de heldere uitleg en snelle reactie van Magmodules kon mijn vraag, en dus mijn Magento-probleem, binnen enkele uren worden beantwoord. Doeltreffend, zonder moeilijke termen, gewoon zo als het zou moeten zijn.... Bedankt!
Google 05 Sep 2025
star star star star star
star star star star star
Denis Metzler
To evaluate a provider, it is not enough to consider only the product offered, but also its after-sales service, such as support and troubleshooting. Magmodules has been extremely satisfactory at all levels on multiple occasions and sets the bar at the top when comparing the competition.
Google 02 Sep 2025
star star star star star
star star star star star
Bleijenberg winkelinrichting en materialen
Goed bereikbaar, reageren snel en denken oplossingsgericht. Een aanrader.
Google 30 Jul 2025
star star star star star
star star star star star
Patrick Verduijn
Magmodules biedt plugins aan die van hoge kwaliteit zijn tegen een goede prijs, waar dit bedrijf in uitblinkt is de bereidheid om de zeldzame feedback & problemen met de plugins te willen onderzoeken, mee te willen denken in het debuggen van problemen en goede oplossingen toe te passen. In mijn decennium ervaring met Magento & 3th parties is Magmodules absoluut een uniqum binnen de markt.
Google 25 Jul 2025
star star star star star
star star star star star
Erik de Groot
Magemodules heeft hele sterke Magento extensies en een proactieve support. Al jaren heel erg tevreden over jullie service en producten!
Google 18 Jul 2025
star star star star star
star star star star star
René Zeuner
We are using the Mollie Magento extension from Magmodules. It works excellently without flaws. Very fast, competent and friendly support. Thanks!
Google 30 Jun 2025
star star star star star
star star star star star
R. U.
Erg goed team, reageren snel en duidelijk en hebben met toegang tot onze database erg goed geholpen (eigenlijk een gratis customization).
Google 18 Jun 2025
star star star star star
star star star star star
Hugo de Groot
Uitstekende support! Wij gebruiken o.a. de Rich Snippets Suite extensie voor onze Magento 2 webshop (Hyvä) en hadden een specifieke vraag over structured data op PLP-pagina’s. Binnen no-time kregen we een inhoudelijk en duidelijk antwoord. Zeer prettig contact en goed onderbouwde uitleg. Absoluut een betrouwbare partner voor Magento-extensies!
Google 13 Jun 2025