/ 
WWW-Crawl4AI-0.001
/ WWW::Crawl4AI::Detect

NAME

WWW::Crawl4AI::Detect - service detection and content-quality classification for Crawl4AI

VERSION

version 0.001

SYNOPSIS

use WWW::Crawl4AI::Detect ();

my $sig = WWW::Crawl4AI::Detect::signals($page);
# { js_required => 0, blocked => 1, captcha => 0, thin_html => 0, http_error => 0 }

if ( !WWW::Crawl4AI::Detect::is_good($page) ) {
  my $why = WWW::Crawl4AI::Detect::why_failed($page);  # 'bot_wall_detected'
}

WWW::Crawl4AI::Detect::probe_cloakbrowser('http://localhost:9222');  # 0/1

DESCRIPTION

The classifier that decides whether a normalized page (as produced by WWW::Crawl4AI::Client) is genuinely useful, and the probes that decide which backends to put into the strategy chain. Pure functions; nothing is exported.

signals

Given a normalized page, returns a hashref of boolean signals: js_required, blocked, captcha, thin_html, http_error. Accepts min_markdown => N to override the thin-content threshold ($WWW::Crawl4AI::Detect::MIN_MARKDOWN, default 500).

blocked reflects content fingerprints (Cloudflare / DataDome / "Just a moment" bodies) — not HTTP status, which lives on its own http_error axis.

captcha means the page is captcha-walled, not merely that a captcha widget exists somewhere: a marker in the rendered markdown counts, but a marker found only in the HTML/script markup (an embedded comment-form reCAPTCHA, a Turnstile login box) counts only when the page is also thin — otherwise a content-rich page with an embedded widget is treated as good.

is_good

True when the page passed all checks: success not explicitly false, no soft/hard HTTP failure, and no negative signal.

why_failed

Returns the most specific failure reason as a short token (captcha, bot_wall_detected, js_required, http_NNN, thin_content) or undef when the page is good.

probe_cloakbrowser

True if a CloakBrowser CDP endpoint answers GET /json/version. Query params on the URL (e.g. ?fingerprint=...) are stripped before probing. Pass ua => $lwp and/or timeout => $secs to control the probe.

detect_proxy_env

Returns $ENV{CRAWL4AI_PROXY_URL} or undef.

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/p5-www-crawl4ai/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <torsten@raudssus.de> https://raudss.us/

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.