🔍Auto-DetectWPWordPressSHShopifyDRDrupalJOJoomlaMOMoodlePSPrestaShopHSHubSpotOCOpenCartGHGhost
tutorials

Why Some Sites Can't Be Scanned by a CMS Detector

Bot protection, JS challenges and rate limits stop detectors reading a site — sometimes the platforms' own. What a failed scan actually tells you.

TW
The Websites Detector Team
Published Aug 29, 20266 min read
📖

Paste a URL into any theme detector and sometimes you get nothing back. Not "no CMS found" — an outright failure to read the page.

This is common enough to be worth explaining, and the reasons are more interesting than "the tool is broken." In several cases the sites that block detectors most aggressively are the official homepages of the platforms those detectors exist to identify.

What a detector actually does

Every tool on this site makes a plain HTTP request. It fetches your target URL the way curl would, reads the HTML, and looks for fingerprints.

What it does not do is run a browser. There is no JavaScript engine, no rendering, no cookie jar built up over a session. That keeps scans fast and cheap, and it is fine for the overwhelming majority of sites, because CMS fingerprints live in the server-rendered HTML.

It also means anything that requires a real browser to get past will stop the scan cold.

The four things that block a scan

1. JavaScript challenges

The most common blocker. Services like Cloudflare's Bot Management, and equivalents from other hosts, intercept requests they consider suspicious and return a holding page instead of your content:

Checking your browser before accessing…

That page carries a small JavaScript proof-of-work. A real browser executes it, gets a cookie, and is let through. A plain HTTP client cannot, so it receives the challenge page and an HTTP 403.

Two platform sites that do this, in our testing:

  • drupal.org
  • prestashop.com's demo chooser

Both return a challenge rather than content. A detector cannot identify Drupal from drupal.org, which is a genuinely funny result and entirely expected.

We hit this from the other direction too. Our own site sits behind Hostinger's bot management, and during an audit we found that Google's Lighthouse — a headless browser — was being served the same challenge page. The tool measuring us could not reach us either. Bot protection does not distinguish between crawlers you want and crawlers you do not.

2. Silent hangs

Worse than a refusal, because there is nothing to report.

ghost.org accepts the TCP connection from a non-browser client and then simply never responds. No error, no timeout from its side, no content. The connection stays open.

We found this the hard way: a request to ghost.org pinned one of our detectors for a full minute before we added timeouts. Every detector now carries an explicit timeout — 15 seconds for a page fetch — precisely so a server that refuses to answer cannot hold a scan open indefinitely.

If a tool has no timeout, this failure mode looks like the tool freezing.

3. Outright 403s for non-browser clients

Some servers check the User-Agent header and reject anything that does not look like a browser.

demo.opencart.com does this to every non-browser client. It is OpenCart's own official demo store, and it cannot be scanned by a tool built to detect OpenCart.

This is a blunt filter and easy to work around by spoofing a browser User-Agent — but doing that as a matter of course is poor manners, and it stops working the moment the site adds a real challenge. We identify ourselves honestly and accept the failures.

4. Rate limiting

Scan the same site repeatedly, or scan from a shared IP address, and you may hit a rate limit — usually HTTP 429.

Shared hosting makes this worse than it sounds. Our detectors run from a shared IP, and Shopify's bot protection sometimes rate-limits requests from it based on other tenants' traffic rather than ours. The same scan can succeed and then fail twenty minutes later with nothing changed on our end.

If a scan fails once, retrying a few minutes later genuinely helps. That is not superstition; it is how rate limit windows work.

What a failed scan tells you

A failure is information, even though it is not the information you asked for.

"The site is behind serious bot protection." That itself says something — usually enterprise hosting, a security-conscious team, or a platform with a history of scraping problems.

It does not mean the site is hiding its CMS. Bot protection is applied at the edge, across the whole site, for reasons that have nothing to do with concealing a theme. Do not read a 403 as evasion.

It does not mean the site is broken. It loads perfectly in your browser. Your browser can solve the challenge; the detector cannot.

What to do instead

When a scan fails, you can still get the answer manually — your browser is the tool that gets through.

1. View the source yourself. Open the site normally, right-click, View Page Source, and search for the markers in our source-code field guide. Your browser has already passed whatever challenge blocked the scanner, so you are reading the real HTML.

2. Check /robots.txt directly. It is frequently served without the same protection, and its default disallow rules are platform-shaped — /wp-admin/ for WordPress, /administrator/ for Joomla.

3. Read the response headers. curl -sI https://example.com returns headers even when the body is a challenge page. X-Powered-By, X-Generator and platform-specific headers often survive.

4. Try a subpage. Bot protection is sometimes applied only to the homepage. A blog post or product page may be readable when / is not.

5. Wait and retry. If it is a rate limit rather than a challenge, the window expires.

Why we do not just use a headless browser

The obvious fix is to render every scan in a real browser, solving JavaScript challenges automatically. We have deliberately not done this, for three reasons.

Cost and speed. A plain HTTP fetch takes well under a second. A headless browser render takes several, uses far more memory, and would make a free tool considerably more expensive to run.

It does not actually work any more. Modern bot management fingerprints headless browsers specifically — automation flags, missing APIs, timing signatures. Our own site's WAF challenges headless Chrome while letting plain HTTP requests through, which is the opposite of what you would naively expect. Headless rendering trades one set of failures for another.

It is an escalation. Deliberately defeating a challenge someone deployed on purpose is a different activity from reading a public page. We would rather report an honest failure than build a tool whose value proposition is circumventing site owners' explicit choices.

The honest summary

A CMS detector reads public HTML. When a site chooses not to serve public HTML to non-browser clients, the detector has nothing to work with — and no amount of engineering changes that without crossing into circumvention.

The failure rate is low. The sites where it bites hardest are, entertainingly, often the platform vendors themselves. And when it happens, thirty seconds with View Source will get you the answer, because your browser is already past the gate.

If a scan does fail, try our detector again in a few minutes — and if it keeps failing on a site that loads fine for you, that is a useful signal about the site, not about the tool.