NAME

rebuild-robots-config

SYNOPSIS

perl devel/rebuild-robots-config robots.toml > rebuilt-robots.toml

DESCRIPTION

This is a quick-and-dirty tool for parsing, normalising and updating the robots.toml configuration.

One of the things that it can do is to read the network information and update that. (If the data cannot be parsed, then the existing data will not be overridden.)

Each robot contains metadata of the form

## Header-Field: Some value

The metadata that are handled by this utility are

Network

## Network: http://example.com/some-bot-ip-list.json

This is a URL containing data about what network blocks the robot connects from.

Plain text is assumed to have one IP address or CIDR per line.

JSON is assumed to be follow the common schema

{
  "creationTime": "2026-08-24T14:45:49.000000",
  "prefixes": [
    {
        "ipv6Prefix": "2001:4860:4801:2008::/64"
    },
    {
        "ipv4Prefix": "74.125.219.96/27"
    }
  ]
}

However, the means of parsing this can be overridden by setting "Query".

It may be specified multiple times.

Query

## Query: .prefixes[] | .ipv4Prefix // .ipv6Prefix

This is a query to feed into JQ::Lite for querying JSON data.

Date

## Date: 2024-08-24

This is the date that the network information was last updated.

About

## About: https://example.com

This is a URL with general information about the robot.

See-Also

## See-Also: https://example.com/list-of-bots-in-html-that-cannot-be-parsed

This contains additional reference information.

Note

## Note: This is a user-readable comment explaining something.

This is a comment or note.

SECURITY CONSIDERATIONS

This utility will attempt to read information about the robots from and update the configuration. That involves retrieving files from websites, parsing them, and updating the configuration based on their contents.

Please verify that the sources of information are trustworthy and that the configuration is not broken by updates.

AUTHOR

Robert Rothenberg <perl@rhizomnic.com>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2018-2026 by Robert Rothenberg.

This is free software, licensed under:

The Artistic License 2.0 (GPL Compatible)