NAME
Gungho::Component::RobotsMETA - Automatically Parse Robots META
SYNOPSIS
components:
- RobotsMETA
DESCRIPTION
This module automatically parses any text/html document for robots exclusion directies embedded in the document.
METHODS
setup
Initializes the component.
handle_response
Overrides Gungho::Component::Core::handle_response()