/ 
WWW-Crawl4AI-0.005
/ WWW::Crawl4AI::Detect

NAME

WWW::Crawl4AI::Detect - service detection and content-quality classification for Crawl4AI

VERSION

version 0.005

SYNOPSIS

use WWW::Crawl4AI::Detect ();

my $sig = WWW::Crawl4AI::Detect::signals($page);
# { blocked => 1, captcha => 0, thin_html => 0, http_error => 0 }

if ( !WWW::Crawl4AI::Detect::is_good($page) ) {
  my $why = WWW::Crawl4AI::Detect::why_failed($page);  # 'bot_wall_detected'
}

WWW::Crawl4AI::Detect::probe_cloakbrowser('http://localhost:9222');  # 0/1

DESCRIPTION

The classifier that decides whether a normalized page (as produced by WWW::Crawl4AI::Client) is genuinely useful, and the probes that decide which backends to put into the strategy chain. Pure functions; nothing is exported.

signals

Given a normalized page, returns a hashref of boolean signals: blocked, captcha, thin_html, http_error. Accepts min_markdown => N to override the thin-content threshold ($WWW::Crawl4AI::Detect::MIN_MARKDOWN, default 500).

Master rule: content volume decides. A bot-wall, a JS shell, and a captcha gate all replace the page content, so they are thin by construction — thin_html catches them. A content-rich page (>= min_markdown chars) that returned 200 is the scrape, and nothing in its body text or <title> may discard it. As of 0.005 there are no body/title phrase heuristics: they were either redundant (on a thin page thin_html already fails it) or outright false-positives (a full article carrying Cloudflare's passive __cf_ beacon in its markup, or a legit "Access denied" <title>, was wrongly thrown away). The only size-independent block signals left are the ones a real content page can never carry.

blocked — the post-redirect final_url is a known WAF challenge endpoint (/cdn-cgi/challenge, __cf_chl, /challenge-platform/, datadome, geo.captcha-delivery.com, /px/captcha, perimeterx): the request left the origin and landed on the gate. Not HTTP status — that lives on the http_error axis, so a bare 403 reads http_403.

captcha — the post-redirect final_url is a captcha provider's own verification endpoint (google.com/recaptcha, /recaptcha/api, hcaptcha.com).

A site that soft-blocks by serving one identical interstitial for every URL (200, no redirect, no challenge final_url) is invisible to per-page signals by design — it is caught one level up, by the caller comparing markdown across the fetched pages.

is_good

True when the page passed all checks: success not explicitly false, no soft/hard HTTP failure, and no negative signal.

why_failed

Returns the most specific failure reason as a short token (captcha, bot_wall_detected, http_NNN, thin_content) or undef when the page is good.

probe_cloakbrowser

True if a CloakBrowser CDP endpoint answers GET /json/version. Query params on the URL (e.g. ?fingerprint=...) are stripped before probing. Pass ua => $lwp and/or timeout => $secs to control the probe.

detect_proxy_env

Returns $ENV{CRAWL4AI_PROXY_URL} or undef.

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/p5-www-crawl4ai/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de> https://raudss.us/

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.