Robots Configuration for SEO
Complete settings reference for the Robots Meta Tag functionality in the Google Sitemap & Robots extension. This guide covers how to control search engine indexing behavior for your pages.
Overview
Navigate to Magmodules → Robots to access robots settings.
The robots module lets you set default meta robots tags for different page types. This controls whether search engines index pages and follow links on them.
General
| Setting | Description |
|---|---|
| Enabled | Enable or disable the robots module |
Understanding Robots Tags
Available Options
| Tag | Meaning |
|---|---|
| INDEX,FOLLOW | Index this page, follow all links |
| NOINDEX,FOLLOW | Don't index this page, but follow links |
| INDEX,NOFOLLOW | Index this page, don't follow links |
| NOINDEX,NOFOLLOW | Don't index, don't follow links |
When to Use Each
INDEX,FOLLOW (default for most pages)
- Product pages you want in search results
- Category pages with unique content
- Important CMS pages (about us, contact)
NOINDEX,FOLLOW
- Filtered category pages (price, color, size filters)
- Search results pages
- Paginated pages (page 2, 3, etc.)
- Thank you / confirmation pages
INDEX,NOFOLLOW
- Pages with user-generated links you don't trust
- Rarely needed in e-commerce
NOINDEX,NOFOLLOW
- Admin or internal pages
- Duplicate content pages
- Staging/test pages
Settings
Set default robots meta tags for different page types:
Category Pages
| Setting | Description |
|---|---|
| Category Pages - Default Robots | Default robots tag for all category pages |
Recommended: INDEX,FOLLOW
Categories are important for SEO. Only use NOINDEX for filtered views or thin content categories.
Product Pages
| Setting | Description |
|---|---|
| Product Pages - Default Robots | Default robots tag for all product pages |
Recommended: INDEX,FOLLOW
Product pages should be indexed. Use product-level overrides for items you want to exclude.
CMS Pages
| Setting | Description |
|---|---|
| CMS Pages - Default Robots | Default robots tag for all CMS pages |
Recommended: INDEX,FOLLOW
Most CMS pages should be indexed. Override individually for pages like "Terms & Conditions" if needed.
Debug
| Setting | Description |
|---|---|
| Debug Mode | Enable detailed logging for robots settings |
Enable when troubleshooting robots tag issues. Check var/log/ for output.
Page-Level Overrides
The extension sets defaults, but you can override per page:
Categories
- Edit category in Catalog → Categories
- Go to Search Engine Optimization section
- Set Robots field to override default
Products
- Edit product in Catalog → Products
- Go to Search Engine Optimization section
- Set Robots field to override default
CMS Pages
- Edit page in Content → Pages
- Go to Search Engine Optimization section
- Set Robots field to override default
Common Scenarios
Scenario 1: Hide Filtered Category Pages
You want main category pages indexed but not filtered views.
Configuration:
- Default Category Robots: INDEX,FOLLOW
- Use layered navigation that adds NOINDEX to filtered URLs
Scenario 2: Exclude Out-of-Stock Products
You don't want out-of-stock products in search results.
Approach:
- Keep default as INDEX,FOLLOW
- Use product attribute or automation to set NOINDEX on out-of-stock items
Scenario 3: New Store Launch
You're preparing a store but don't want it indexed yet.
Configuration:
- Set all defaults to NOINDEX,NOFOLLOW
- Change to INDEX,FOLLOW when ready to launch
Integration with Sitemap
The robots module works together with the sitemap module:
- Pages marked NOINDEX can be excluded from sitemap automatically
- Enable "Exclude No-Index" options in sitemap settings
- This prevents sending mixed signals to search engines
Verifying Robots Tags
Check in Browser
- Open your page
- View page source (Ctrl+U or Cmd+U)
- Look for:
<meta name="robots" content="INDEX,FOLLOW">
Using Google Search Console
- Go to URL Inspection
- Enter your page URL
- Check "Indexing allowed?" status
Need More Help?
Documentation:
- All Help Articles - Complete documentation overview
Support:
- Contact Support - Get help from our team