Skip to guide

How to protect your website from AI scraping

Decide what can be public, put sensitive content behind server-side access checks, and limit how much data a client can collect from protected routes.

Protect the data at its source

Start with the content you want to protect. A catalog might have public product descriptions, an account-only inventory feed and a large export endpoint. Keep the product pages available for discovery. Require authorization on both the inventory feed and the export, then check that direct downloads and cached copies follow the same access rules.

Stopnetic evaluates requests at the application server before guarded responses are sent. Use route rules alongside your application’s permissions. For a client preview, you can require a shared password before serving protected pages.

Limit repeated and bulk access

Use browser verification where an extra check is appropriate and automation checks to block signals the guard detects. Some automated browsers can complete browser verification, so use separate access controls for valuable data.

A request budget limits repeated calls. A data budget can account for how many records the server returns, which matters if one call can download hundreds of results. Start with the request budget setup and consider data budgets for larger responses. Both require a compatible integration and trusted network configuration.

Test the rule from a visitor’s perspective

Open the public page signed out, call the protected endpoint without authorization, and test a permitted request with a valid session. Confirm the actual response and check the decision reason in Event explorer. If a budget changes access, verify that normal usage stays within the allowance.

Any content you serve publicly can be copied. Use access checks for content that needs to stay private and reserve crawler directives for communicating preferences.

For public pages you want indexed, see how to manage web crawlers without blocking Google.