Skip to guide

How to manage web crawlers without blocking Google

Keep the pages you want in search open to crawlers. Apply server-side controls to routes that expose private data, expensive queries or bulk downloads.

Choose which URLs belong in search

List your public pages, such as product pages, documentation and articles. Include their preferred URLs in your sitemap. A signed-out visitor and Googlebot should be able to load those pages and the resources needed to render them without a password or a browser challenge.

Publish a robots.txt file at the root of the site to communicate crawling preferences and point crawlers to your sitemap. Crawlers may ignore those preferences, so use authentication for private data and check access before your server returns it. See Google’s guide to robots.txt.

Apply protection to specific routes

For a site with public documentation and a customer export API, keep the docs public and require application authorization on the export endpoint. If repeated access is the problem, add a request limit to that endpoint. Stopnetic supports route rules and request budgets on compatible server setups.

Stopnetic evaluates route rules in order. Put specific public exceptions above broader rules and confirm how your site handles APIs, assets and other hostnames. See the page and API setup guide.

Check what Google and your logs actually see

Run a live URL test in Search Console after changing your policies. Check that Google can fetch the full public page with HTTP 200 and that the page allows indexing. Review Stopnetic’s Event explorer for the reason behind a challenge or block, and compare it with your hosting logs for the final response.

A request to /robots.txt is common for crawlers. Treat it as a clue about the request, since the path alone cannot identify the caller.

If the concern is bulk extraction, read the guide to protecting pages and APIs from AI scraping.