Crumpled.RobotsTxt

A flexible, configuration-driven robots.txt solution for Umbraco v13, v14, v15, v16 & v17

Repository Structure

This repository contains:

• Crumpled.RobotsTxt - Main Umbraco package for managing robots.txt configuration (README)
• Crumpled.RobotsTxt.Core - Internal ASP.NET Core robots.txt middleware implementation
• Crumpled.RobotsTxt.TestSite - Test Umbraco site for development (Umbraco v17.*)
• Crumpled.RobotsTxt.TestSite13 - Test Umbraco site for v13 compatibility

Installation

dotnet add package Crumpled.RobotsTxt

📖 View Full Documentation & Configuration Guide

Key Features

• 🛡️ Safe by Default - Blocks all bots by default to prevent accidental indexing
• 🌍 Multi-Site & Environment-Aware - Different rules per domain/environment
• 📝 Flexible Configuration - Reusable rulesets via appsettings.json
• 🗺️ Sitemap Integration - Automatic sitemap URL generation
• ⚙️ Zero Code Setup - Auto-registration via Umbraco Composer

Development

Test Sites

Crumpled.RobotsTxt.TestSite - Umbraco v17 test site with unattended installation. Credentials (not that you really need them) are set in appsettings.Development.json

Crumpled.RobotsTxt.TestSite13 - Umbraco v13 test site for backward compatibility testing

Launch Profiles

The main test site includes three launch profiles:

1. Crumpled.RobotsTxt.TestSite (Development Profile)

• Tests multi-site robots.txt functionality
• Listens on three ports simultaneously:
  • https://localhost:44389 - "Stage" site (Development ruleset)
  • https://localhost:44390 - "Prod" site (Production ruleset)
  • https://localhost:44391 - Unmatched domain (tests fallback behavior)
• Each URL serves different robots.txt content based on hostname configuration

2. Crumpled.RobotsTxt.TestSiteLiveCloud (Umbraco Cloud Live Simulation)

• Tests Umbraco Cloud live environment detection
• Sets UMBRACO__CLOUD__DEPLOY__ENVIRONMENTNAME=live
• Single URL: https://localhost:44392
• Environment: CloudTest
• Demonstrates Cloud-specific default behavior (allows all bots when no sites configured)

3. Crumpled.RobotsTxt.TestSiteDevCloud (Umbraco Cloud Dev Simulation)

• Tests Umbraco Cloud development environment detection
• Sets UMBRACO__CLOUD__DEPLOY__ENVIRONMENTNAME=development
• Single URL: https://localhost:44393
• Environment: CloudTest
• Demonstrates Cloud dev environment behavior (blocks all bots by default)

Running the Test Sites

Run with the default profile:

dotnet run --project src/Crumpled.RobotsTxt.TestSite

Or specify a launch profile:

dotnet run --project src/Crumpled.RobotsTxt.TestSite --launch-profile Crumpled.RobotsTxt.TestSiteLiveCloud
dotnet run --project src/Crumpled.RobotsTxt.TestSite --launch-profile Crumpled.RobotsTxt.TestSiteDevCloud

Run the Umbraco v13 test site:

dotnet run --project src/Crumpled.RobotsTxt.TestSite13