NAME

Database::BI::Model::Geocode - Resolve a location column to coordinates, once per distinct value

VERSION

0.020.0

SYNOPSIS

Geocode a table's place column and build map markers:

use Database::BI::Model::Geocode;

my $geocode = Database::BI::Model::Geocode->new();
my $result  = $geocode->resolve_column($rows, 'city');

for my $marker (@{ $result->{'markers'} }) {
    printf "  %s: lat=%.4f lon=%.4f (%d rows)\n",
        $marker->{'value'}, $marker->{'lat'}, $marker->{'lon'},
        $marker->{'rows'};
}

my $c = $result->{'counts'};
printf "Plotted %d of %d rows (%d skipped, %d unresolved)\n",
    $c->{'plotted'}, $c->{'rows'}, $c->{'skipped'}, $c->{'unresolved'};

Re-use remembered answers so the next request costs nothing:

# Load what was stored from a previous run.
my $spec   = 'table:sales';
my $models = decode_json(Mojo::File->new('.bi/models.json')->slurp);
my $cache  = Database::BI::Model::Geocode->cache_from_models($models, $spec);

# Resolve; cached values are used without asking the geocoder.
my $result = $geocode->resolve_column($rows, 'city', cache => $cache);

# Save the updated cache back to the models file.
my $updated = Database::BI::Model::Geocode->models_with_cache(
    $models, $spec, $result->{'cache'},
    backend => $result->{'backend'},
    built   => '2026-10-18T12:00:00',
);
Mojo::File->new('.bi/models.json')->spew(encode_json($updated));

Warm the cache for a whole column without needing rows:

# Extract unique city values from your own data.
my @cities = do { my %s; grep { !$s{$_}++ } map { $_->{'city'} } @{ $rows } };

my $result = $geocode->resolve_values(\@cities,
    cache  => $cache,
    region => 'GB',
    max    => 1000,
);

printf "Resolved %d / %d new places\n",
    $result->{'counts'}{'resolved'}, $result->{'counts'}{'attempted'};

Ask whether the geocoder is available before offering it in the UI:

my $geocode = Database::BI::Model::Geocode->new();
my ($ok, $why) = $geocode->geocoder_available();

unless ($ok) {
    carp "Map panel disabled: $why";
}

Inject a test double so geocoding works without a real geocoder:

package MyGeocoderStub;
sub new            { bless {}, shift }
sub valid_location { my ($self, $v) = @_; $v && $v !~ /\A\d+\z/ }
sub geocode_many   {
    my ($self, $places, %opts) = @_;
    return { 'London' => { lat => 51.5, long => -0.12, database => 'stub' } };
}

my $geocode = Database::BI::Model::Geocode->new(geocoder => MyGeocoderStub->new);
my $result  = $geocode->resolve_column($rows, 'city');

DESCRIPTION

The geocoder is a Geo::Coder::List chain:

Geo::Coder::List->new()
    ->push(Geo::Coder::Free->new())   # the local data
    ->push(Geo::Coder::OSM->new());   # OpenStreetMap's Nominatim

Geo::Coder::Free answers from data on this machine. A place it does not know is asked of Nominatim, over the network, through an LWP::UserAgent::Throttled that keeps Nominatim's usage policy: one request a second, from a client that names itself (Database-BI/version), with a ten-second time-out. The user agent is shared by every lookup in the process, so the gap holds across requests. Nominatim is told the place name and nothing else.

A table column holds something less tidy than a place name: values repeated thousands of times, blank cells, numbers that drifted in from an adjacent column. This module does the three things the column then needs.

Ask once per distinct value.

The column is reduced to a set before any lookup happens. A repeated city name costs one lookup, so a 100,000-row file with a dozen cities costs twelve lookups, not 100,000.

Refuse a value that is not a location before any backend sees it.

$geocoder->valid_location($value) is the gate the geocoder's own guard applies before looking anything up. This module asks the same question first, so a pure-digit string (a row number that ended up in the wrong column), an empty cell, or a Perl reference is counted and reported as skipped rather than sent to a backend that would reject it anyway.

Look the rest up without letting one value end the batch.

Each place goes down a Geo::Coder::List on its own: the local data, then Nominatim when it is in the chain. With the local data alone the list holds only Geo::Coder::Free, which is given the region hint (Geo::Coder::OSM would put it in the Nominatim query). Either way the list's cache answers what it has found before. Geo::Coder::List catches what a geocoder throws and asks the next one, so a throw does not end the request; a place whose lookup threw and found nothing is held (put in not_attempted, not cached), because "not found" is remembered for good and an error is not an answer.

An injected geocoder is asked for one geocode_many() batch instead, requested with on_error => 'undef'. A value it refuses after the pre-filter is a miss; a backend that throws is still a throw, because nothing here wraps it in an eval.

Tell a Nominatim that failed from a place it does not know.

Geo::Coder::OSM answers nothing in both cases. The chain's log says whether Nominatim was asked, and its last response says whether it answered. A place it failed on is held, not remembered as not found, and Nominatim is not asked again in that request: the rest are asked of the local data alone, and the ones it does not know are held too. The page is told Nominatim's status line and how many places wait, never which.

The counts the module returns add up, always.

skipped + cached + attempted + not_attempted  ==  distinct
resolved + unresolved                         ==  cached + attempted

lookups is the number of places that reached a geocoder rather than the chain's own cache. For an injected geocoder it is the number that geocoder reports for the batch, which Geo::Coder::Free 0.48 and later keeps lower than attempted when a value is answered from a warm process cache. It never exceeds attempted. online is how many of them were asked of Nominatim.

The cache

A cache entry is a [lat, lon] arrayref for a place that resolved, or the scalar 0 for a place that was looked up and not found. That 0 sentinel prevents a column of unresolvable values from being retried on every page request.

The cache is passed per call and returned per call. The caller's hashref is never modified: a request that dies part way through cannot leave a half-written cache behind. The controller is responsible for persisting the returned cache, which keeps the model_dir-inside-data_dir guard in the one place that already owns it.

A previously stored cache is used even when the geocoder is not available. Places already resolved continue to be plotted; only new lookups are unavailable.

Without the geocoder

Geo::Coder::Free and Geo::Coder::List are recommended dependencies, not required ones. They are loaded the first time they are needed, not at compile time. Geo::Coder::OSM and LWP::UserAgent::Throttled are recommended too; without them the chain is the local data alone, and a request with places it did not know says so (warn_map_geocoder_local_only). When Geo::Coder::Free or Geo::Coder::List is absent or older than its minimum version:

$geocode->geocoder_available() returns (0, $reason).
resolve_column and resolve_values still return a complete result: all values are placed in not_attempted, cached answers are still plotted, and the error_map_geocoder_missing message explains the situation.
Nothing croaks. The page renders with whatever the cache already holds.

CONSTRUCTOR

new

my $geocode = Database::BI::Model::Geocode->new(%options);
my $geocode = Database::BI::Model::Geocode->new(\%options);  # hashref also accepted

Create a new resolver. Accepts a flat key/value list or a hashref; both forms are handled by Params::Get.

API SPECIFICATION

INPUT

geocoder             => object,   # optional; duck-typed: must provide valid_location() and geocode_many()
openaddr             => string,   # optional; path to OpenAddresses data (default: $ENV{OPENADDR_HOME})
min_geocoder_version => string,   # optional; minimum version to accept (default: '0.48')
i18n                 => object,   # optional; must implement maketext($key, @args)
online               => boolean,  # optional; ask Nominatim about places the local data does not know (default: 1)
online_max           => positive integer, # optional; places asked of Nominatim per call (default: no limit)
cache                => hashref|object, # optional; Geo::Coder::List's cache (a hashref, or an object with get() and set(), such as file_cache()); default: one CHI Memory cache per process

geocoder is the injection seam for tests. An injected geocoder is used exactly as given: it is not version-checked, so any object that responds to valid_location() and geocode_many() works. A geocoder that fails either can() check causes an immediate croak (error_geocoder_interface), not a silent degradation.

openaddr overrides $ENV{OPENADDR_HOME}. It is forwarded to Geo::Coder::Free->new(openaddr => ...). When it is absent and the environment variable is not set, the geocoder uses its built-in MaxMind data only, which resolves city and country names but not street addresses. warn_map_geocoder_unavailable_data is emitted in that case.

min_geocoder_version lets the caller pin the floor, or a test simulate a too-old installation on a machine that has the current version.

online => 0 keeps every lookup on this machine: a place the local data does not know is then not sent to OpenStreetMap. online_max caps how many places one resolve_*() call asks Nominatim about. At one a second, it is roughly how long the call can wait for Nominatim; the places past it go into not_attempted uncached, so the next call asks them. An injected geocoder is used as it is, and these two do not apply to it.

i18n is an optional object whose maketext($key, @args) method is called in place of the built-in %MESSAGES table. The key is one of the constants defined in %MESSAGES; the args are the same positional values sprintf would receive.

OUTPUT

Returns $self.

Croaks with error_geocoder_interface if the injected geocoder does not provide both valid_location() and geocode_many(), and with error_online_max_invalid if online_max is not a positive integer.

Never croaks because the geocoder module is absent or too old: that is a runtime condition reported by geocoder_available(), not a constructor error.

SUBROUTINES/METHODS

resolve_column

my $result = $geocode->resolve_column(\@rows, $column_name, %options);

Resolve one named column of a table. \@rows is an arrayref of hashrefs (the records from the data pipeline), and $column_name is the key to look up in each row.

This is the main entry point for the map controller. It runs the same resolution as "resolve_values", then adds per-marker row counts and the markers list that a map rendering loop needs.

API SPECIFICATION

INPUT

\@rows          => arrayref of hashrefs, required
$column_name    => non-empty string, required
cache    => hashref,            # optional; { value => [lat, lon] | 0 }
region   => two-letter string,  # optional; country hint for the geocoder (e.g. 'GB')
max      => positive integer,   # optional; max new lookups; default 500
lookup   => 0 | 1,              # optional; 0 asks nothing and lists what it would ask; default 1

\@rows is the full record set as returned by the pipeline. Rows that are not hashrefs produce an undef cell, which is classified as skipped.

$column_name must be a non-empty string. It is used as a hash key; it is not validated against a schema, so a column name with spaces or special characters is accepted if the data uses one.

cache holds the already-known answers for this column. Entries are either a [lat, lon] arrayref (resolved) or 0 (looked up and not found). The caller's hashref is never modified; the returned cache key holds the updated copy.

region is passed straight through to the geocoder as a country hint. It is not derived from the Accept-Language header, because a language is not a country.

max is counted in new lookups, not rows. A cached answer costs nothing and does not count toward it. Values held back by the cap go into not_attempted and are not stored in the cache, so the next request picks them up.

lookup => 0 plots from cache alone and asks the geocoder nothing. The places it would have asked (up to max, in first-seen order) are returned in pending, and are counted in not_attempted meanwhile. A caller can then look them up one at a time with "resolve_values", saving each answer as it comes, which is how the map page shows its progress and can be cancelled. The geocoder is still loaded, because its rule decides which values are places.

OUTPUT

Returns a hashref:

{
    available     => 1 | 0,
    points        => { $value => { lat => $n, lon => $n, rows => $n, first_row => $n, backend? => $s } },
    markers       => [ { value => $s, lat => $n, lon => $n, rows => $n, first_row => $n } ],
    counts        => {
        rows          => $n,   # total rows in \@rows
        plotted       => $n,   # rows whose value resolved to a point
        unusable      => $n,   # rows - plotted
        distinct      => $n,   # unique values seen
        skipped       => $n,   # refused before any lookup (empty, ref, digit-only)
        cached        => $n,   # found in the cache (resolved or not)
        attempted     => $n,   # sent to the geocoder this request
        lookups       => $n,   # lookups the geocoder reports (may be < attempted)
        online        => $n,   # of those, places asked of Nominatim
        resolved      => $n,   # total with a position (cache + new)
        unresolved    => $n,   # total without (cache + new)
        not_attempted => $n,   # held back by a cap, by a failed lookup, or because no geocoder
    },
    skipped       => [ $value, ... ],   # values refused before lookup
    unresolved    => [ $value, ... ],   # values tried (now or before) and not found
    not_attempted => [ $value, ... ],   # held back by the cap
    pending       => [ $value, ... ],   # with lookup => 0: what would have been asked; else empty
    messages      => [ { key => $k, text => $t }, ... ],
    cache         => { $value => [lat, lon] | 0 },   # updated copy; never the caller's hashref
    backends      => { $backend_name => $count },
    backend       => $backend_name | undef,          # undef when nothing resolved
    capped        => 0 | 1,
}

points holds one entry per value that produced a position, whether the position came from the cache or from this request. markers is the same data in first-seen row order, which reads better on a map than alphabetical order.

backend on a point entry is set only for values looked up in this request; cached values do not carry one because the cache stores a position, not which database gave it. The top-level backend is the name of the backend that answered the most lookups this request.

The count invariant always holds:

skipped + cached + attempted + not_attempted  ==  distinct

MESSAGES

error_rows_arrayref                -- \@rows is not an arrayref
error_column_required              -- $column_name is empty or undef
error_map_loc_column               -- column not found in any row
error_map_geocoder_missing         -- no geocoder, or version below the floor
error_map_geocoder_list_missing    -- Geo::Coder::List absent or too old
warn_map_rows_skipped              -- some rows could not be plotted, and why
warn_map_rows_why_*                -- the reasons it names (not messages of their own)
warn_map_geocode_capped            -- max or online_max held some values back
warn_map_geocoder_unavailable_data -- local data only, and OPENADDR_HOME not set
warn_map_geocoder_local_only       -- no Geo::Coder::OSM, and places were not found
warn_map_geocoder_online_failed    -- Nominatim did not answer; places held

EXAMPLE

my $cache   = Database::BI::Model::Geocode->cache_from_models($models, $spec);
my $result  = $geocode->resolve_column($rows, 'city',
    cache  => $cache,
    region => 'GB',
    max    => 200,
);

if ($result->{'capped'}) {
    carp "Some cities were not looked up; load the page again to finish them.";
}

for my $m (@{ $result->{'markers'} }) {
    printf "[%.4f, %.4f]  %s (%d rows)\n",
        $m->{'lat'}, $m->{'lon'}, $m->{'value'}, $m->{'rows'};
}

resolve_values

my $result = $geocode->resolve_values(\@values, %options);
my $result = $geocode->resolve_values(\@values, \%options);

Resolve a flat list of cell values, without the context of any row structure. Use this when you want to warm the cache for a column's distinct values, or when you have already extracted the values yourself and do not need per-marker row counts.

The returned hashref is identical to "resolve_column"'s, except that counts.rows, counts.plotted, counts.unusable, and markers are absent, and warn_map_rows_skipped is never emitted.

API SPECIFICATION

INPUT

\@values => arrayref, required;  elements may be any scalar or undef
cache    => hashref,             # optional; { value => [lat, lon] | 0 }
region   => string,              # optional; two-letter country hint
max      => positive integer,    # optional; new lookups allowed; default 500
lookup   => 0 | 1,               # optional; 0 asks nothing (see resolve_column); default 1

Duplicate values in \@values cost one lookup each. undef, references, empty strings, and pure-digit strings are refused before any lookup and counted as skipped.

OUTPUT

Returns a hashref (same shape as "resolve_column" but without markers, counts.rows, counts.plotted, and counts.unusable):

{
    available     => 1 | 0,
    points        => { $value => { lat => $n, lon => $n, backend? => $s } },
    counts        => {
        distinct      => $n,
        skipped       => $n,
        cached        => $n,
        attempted     => $n,
        lookups       => $n,
        resolved      => $n,
        unresolved    => $n,
        not_attempted => $n,
    },
    skipped       => [ $value, ... ],
    unresolved    => [ $value, ... ],
    not_attempted => [ $value, ... ],
    pending       => [ $value, ... ],
    messages      => [ { key => $k, text => $t }, ... ],
    cache         => { $value => [lat, lon] | 0 },
    backends      => { $name => $count },
    backend       => $name | undef,
    capped        => 0 | 1,
}

Croaks with error_values_arrayref if \@values is not an arrayref.

EXAMPLE

# Build the distinct-value list yourself and warm the cache offline.
my %seen;
my @cities = grep { defined($_) && !$seen{$_}++ } map { $_->{'city'} } @{ $rows };

my $result = $geocode->resolve_values(\@cities,
    cache  => $cache,
    region => 'GB',
    max    => 2000,
);

printf "New lookups: %d  Resolved: %d  Unresolved: %d\n",
    $result->{'counts'}{'attempted'},
    $result->{'counts'}{'resolved'},
    $result->{'counts'}{'unresolved'};

resolve_column

See full documentation above, in "SUBROUTINES/METHODS".

geocoder_available

my ($is_available, $reason) = $geocode->geocoder_available();

Ask whether this object can make new geocoder lookups. Returns a list of two values: a boolean (1 or 0) and a human-readable reason string when the answer is no.

Loads Geo::Coder::Free lazily on the first call if it has not been loaded yet. Never croaks: a panel uses this to decide whether to show the location option, so the answer must always be a string, not an exception.

API SPECIFICATION

INPUT

None. (No arguments accepted.)

OUTPUT

($available, $reason)

$available => 1      # geocoder is ready
$available => 0      # geocoder is not available
$reason    => undef  # when $available is 1
$reason    => string # error_map_geocoder_missing text; when $available is 0

EXAMPLE

my ($ok, $reason) = $geocode->geocoder_available();

if (!$ok) {
    # Disable the "Map by location column" panel option.
    $stash->{'map_loc_disabled'} = 1;
    $stash->{'map_loc_disabled_reason'} = $reason;
}

file_cache

my $cache   = Database::BI::Model::Geocode->file_cache($model_dir);
my $geocode = Database::BI::Model::Geocode->new(cache => $cache);

The cache Geo::Coder::List keeps its answers in, as files under $model_dir/geocode-cache: a place Nominatim found is not asked again for a month (Geo::Coder::List's cache_hit_duration), across requests, restarts and every hypnotoad worker. A copy of each answer is also kept in the process's memory. The directory is made when the first answer is stored.

This is a class method. The map page and script/bi-geocode both call it with the directory that holds models.json, so they share one cache. It holds place names, as models.json does, so it belongs in model_dir and never in data_dir.

API SPECIFICATION

INPUT

$model_dir => string, required; an existing directory

OUTPUT

A CHI object (the File driver), for the cache option of new(). Croaks with error_cache_dir when $model_dir is not a directory.

EXAMPLE

my $geocode = Database::BI::Model::Geocode->new(
    cache => Database::BI::Model::Geocode->file_cache('.bi'),
);

place_rows

my ($places, $name) = Database::BI::Model::Geocode->place_rows(\@rows, [ 'Address', 'Country' ]);
my $result = $geocode->resolve_column($places, $name);

A place written across several columns, such as an address and then its country, as one column to geocode. Each row's cells are trimmed and joined in the order given, with ", ", leaving out the empty ones: "Brezje 8, Cerknica" and "Slovenia" become "Brezje 8, Cerknica, Slovenia". A row with every cell empty gives '', which resolve_column counts as skipped.

The rows themselves are not changed (a SQL backend's are locked): the result is a new row for each, in the same order, holding only the joined place, so a marker's first_row still indexes \@rows. With a single column the rows are returned as they are, under that column's name.

API SPECIFICATION

INPUT

\@rows       arrayref of hashrefs        # required; a row that is not a hashref gives undef
\@columns    arrayref of column names    # required; one or more, in order

OUTPUT

(\@places, $name)
    \@places   one hashref { $name => $place } per row, or \@rows itself for one column
    $name      the columns joined with ", " ("Address, Country")

Croaks with error_rows_arrayref when \@rows is not an arrayref, and with error_column_required when \@columns names no column.

EXAMPLE

# loc=Address&loc=Country on the map page
my ($places, $name) = Database::BI::Model::Geocode->place_rows($records, [qw(Address Country)]);
my $result = Database::BI::Model::Geocode->new->resolve_column($places, $name, cache => \%cache);

cache_from_models

my $cache = Database::BI::Model::Geocode->cache_from_models(\%models, $spec_key);

Extract the geocode cache for one dataset spec from a model store. Returns a copy of the stored places hashref so the caller cannot accidentally modify the model store.

This is a class method: call it on the package name, or on an instance if you prefer; either way the invocant is ignored.

API SPECIFICATION

INPUT

\%models  => hashref, required;  the structure returned by _read_models()
$spec_key => string, required;   the dataset spec (e.g. 'table:sales')

OUTPUT

Returns a plain hashref:

{ $place_name => [lat, lon] | 0, ... }

Returns an empty hashref {} for an unknown spec, a spec that has never been geocoded, or a \%models argument that is not a hashref. Never croaks.

EXAMPLE

my $models = decode_json(Mojo::File->new($model_path)->slurp);
my $cache  = Database::BI::Model::Geocode->cache_from_models(
    $models, 'table:sales'
);

# $cache is now { 'Ramsgate' => [51.336, 1.416], 'Paris' => 0, ... }

models_with_cache

my $updated_models = Database::BI::Model::Geocode->models_with_cache(
    \%models, $spec_key, \%cache,
    backend => 'OpenAddresses',
    built   => '2026-10-18T12:00:00',
);

Produce a new model store in which the geocode cache for $spec_key is replaced with \%cache. Every other key in the model store, and every other spec's geocode data, is carried over unchanged.

This is a class method; the invocant is ignored.

This module never writes the file. The caller is responsible for passing $updated_models to _write_models() or its equivalent, which is where the writability and data_dir guards live. The separation keeps those guards in the one place that already owns them

API SPECIFICATION

INPUT

\%models  => hashref, required;  the existing model store (not modified)
$spec_key => string, required;   the dataset spec to update
\%cache   => hashref, required;  { place => [lat, lon] | 0 }
backend   => string, optional;   name of the geocoder backend that resolved
built     => string, optional;   ISO-8601 timestamp of when the cache was built

OUTPUT

Returns a new hashref. The caller's \%models is never modified.

{
    ...all existing model keys...,
    geo => {
        ...all existing geo specs...,
        $spec_key => {
            places  => \%cache,
            backend => $backend,  # if supplied
            built   => $built,    # if supplied
        },
    },
}

EXAMPLE

my $updated = Database::BI::Model::Geocode->models_with_cache(
    $models,
    'path:/data/sales.csv',
    $result->{'cache'},
    backend => $result->{'backend'},
    built   => '2026-10-18T09:15:42',
);

# Now write the updated store to disk.
Mojo::File->new($model_path)->spew(encode_json($updated));

shared_cache_from_models

my $cache = Database::BI::Model::Geocode->shared_cache_from_models(\%models);

Extract the place answers every dataset shares, as a copy the caller may modify. This is the cache a map request should pass to resolve_column: a place geocoded for one file is then plotted for every other file that names it, without a second lookup.

A model store written before the cache was shared has no shared entry; this returns {} and the per-dataset caches remain readable with cache_from_models. A caller that reads both gets the old answers once and writes them back to the shared entry (see models_with_shared_cache).

This is a class method; the invocant is ignored.

API SPECIFICATION

INPUT

\%models  => hashref, required;  the structure returned by _read_models()

OUTPUT

{ $place_name => [lat, lon] | 0, ... }

Returns an empty hashref for a store with no shared cache, for a malformed entry, or for a \%models argument that is not a hashref. Never croaks.

EXAMPLE

my $models = decode_json(Mojo::File->new($model_path)->slurp);
my $cache  = Database::BI::Model::Geocode->shared_cache_from_models($models);

# $cache is now { 'Ramsgate' => [51.336, 1.416], 'Paris' => 0, ... }

models_with_shared_cache

my $updated_models = Database::BI::Model::Geocode->models_with_shared_cache(
    \%models, \%cache,
    backend => 'OpenAddresses',
    built   => '2026-10-18T12:00:00',
);

Produce a new model store in which the shared place cache holds \%cache. The cache is merged, not replaced: a request writes back every place it saw, and another request may have answered a different place since this one read the store. Other model keys, and every per-dataset geo entry, are carried over unchanged.

This module never writes the file, and does not add a per-dataset entry. The caller passes the result to _write_models() or its equivalent, where the writability and data_dir guards live.

API SPECIFICATION

INPUT

\%models => hashref, required;  the existing model store (not modified)
\%cache  => hashref, required;  { place => [lat, lon] | 0 }
backend  => string, optional;   name of the geocoder backend that resolved
built    => string, optional;   ISO-8601 timestamp of the cache update

OUTPUT

Returns a new hashref. The caller's \%models is never modified.

{
    ...all existing model keys...,
    geo => {
        ...all existing per-dataset entries...,
        shared => {
            places  => \%merged_cache,
            backend => $backend,  # if supplied
            built   => $built,    # if supplied
        },
    },
}

EXAMPLE

my $updated = Database::BI::Model::Geocode->models_with_shared_cache(
    $models,
    $result->{'cache'},
    backend => $result->{'backend'},
    built   => '2026-10-18T09:15:42',
);

Mojo::File->new($model_path)->spew(encode_json($updated));

COMMON PITFALLS

The cache sentinel 0 is not "false -- never looked up".

A cache entry of 0 means the place was looked up and not found. An absent key means it has never been attempted. The module treats these two states differently: a 0 entry contributes to counts.unresolved without triggering a new lookup; an absent key enters the classification loop and may reach the geocoder.

# WRONG -- checking definedness misses the 0 sentinel.
my $resolved = defined($cache{$place});

# CORRECT -- use the internal _has_position logic.
my $resolved = (ref($cache{$place}) eq 'ARRAY');

Pure-digit strings are skipped, not attempted.

A value like '42' fails valid_location() and goes into @skipped. It is never stored in the cache. This is the correct behaviour for a row number that ended up in the wrong column, but it surprises callers who expect every non-empty value to be attempted.

The returned cache is a copy; the caller's hashref is never modified.

Do not compare $cache_you_passed_in with $result-{'cache'}> expecting them to be the same reference. The update is in $result-{'cache'}>.

resolve_column croaks when the column is absent; resolve_values does not.

resolve_column checks that the named column appears in at least one row. If it does not, it croaks with error_map_loc_column. resolve_values does not do this check, because it receives values, not rows.

max counts new lookups, not rows or distinct values.

A value already in cache does not count toward max, even if it was looked up in a previous request. A value skipped by valid_location() does not count either.

new() always succeeds; geocoder load failure is not a constructor error.

You can always call new() without checking whether Geo::Coder::Free is installed. The load failure is deferred to the first call that needs the geocoder, and is reported through geocoder_available() and the messages array in the result, not through an exception.

cache_from_models and models_with_cache are class methods.

You may call them on an instance for convenience, but the invocant is not used. The pattern Database::BI::Model::Geocode->cache_from_models(...) is clearer.

An injected geocoder's batch receives on_error = 'undef'.>

A value an injected geocoder rejects after the valid_location() pre-filter is returned as undef from the batch; it does not croak. A backend that throws an exception (a real database error, a network failure) still propagates upward, because nothing here wraps the geocoder in an eval.

ENCODING

Place names (values in the location column): Passed to the geocoder as byte strings. UTF-8 place names such as "Koeln" or "Beijing" are supported if the geocoder backend handles them. The internal code does not re-encode or validate the bytes; it compares them as opaque strings for deduplication and caching.
Column names: Matched against row hash keys with a string equality test. The column name must be byte-for-byte identical to the key in the row hashrefs. Column names with non-ASCII bytes are accepted.
Spec keys (the $spec_key argument): Used as hash keys in the model store. Any byte sequence is safe; UTF-8 spec keys are stored and retrieved without modification.
Message strings: The built-in %MESSAGES table contains ASCII-only text. If your application needs non-ASCII messages (for example, a French translation), supply an i18n object to new() whose maketext method renders the key in the desired locale.
Geocoder result strings (backend names): Stored and reported as returned by the geocoder. The module does not impose any encoding on them.

DIAGNOSTICS

All user-visible messages are keyed in %Database::BI::Model::Geocode::MESSAGES.

error_map_geocoder_missing

Geocoding a location column needs Geo::Coder::Free %s or later, and %s

The geocoder module is not installed (the second %s is "it is not installed") or is too old (the second %s is the installed version number). The first %s is the floor this module checks for - 0.48 unless the caller pinned min_geocoder_version. Install or upgrade Geo::Coder::Free.

error_map_geocoder_list_missing

Geocoding a location column needs Geo::Coder::List 0.39 or later, and %s

Geo::Coder::Free loaded but Geo::Coder::List, which holds the chain, did not: the %s is "it is not installed" or the version that is. Install or upgrade Geo::Coder::List.

error_cache_dir

Geocode: the geocode cache needs a directory that exists, not "%s"

file_cache() was given a path that is not a directory. Create it first.

error_cache_interface

Geocode: option "cache" must be a hashref or an object with get() and set(), not a %s

new() was given a cache object that Geo::Coder::List cannot use.

error_online_max_invalid

Geocode: option "online_max" must be a positive integer, not "%s"

new() was given an online_max that is not a positive integer. Leave it out for no limit, or pass online => 0 to ask Nominatim nothing.

error_map_loc_column

Column "%s" is not in this dataset, so there is nothing to geocode

The $column_name argument to resolve_column() did not appear as a key in any row. Check the column name, including case and any spaces.

error_geocoder_interface

Geocode: the geocoder must provide %s; Geo::Coder::Free %s or later does

An injected geocoder object does not respond to valid_location() or geocode_many(). Either use the real Geo::Coder::Free or add the missing method to your stub.

error_max_invalid

Geocode: option "max" must be a positive integer, not "%s"

The max option was supplied but is not a positive integer string. Passing max => 0 and max => 'all' both trigger this error.

error_values_arrayref

Geocode: the values must be an arrayref, not "%s"

The first argument to resolve_values() is not an arrayref. Pass \@values (a reference), not @values (a flat list).

error_rows_arrayref

Geocode: the rows must be an arrayref, not "%s"

The first argument to resolve_column() is not an arrayref.

error_column_required

Geocode: the column name is required

The $column_name argument to resolve_column() is undef or an empty string.

warn_map_rows_skipped

Plotted %d of %d rows; %d could not be plotted: %s

Some rows could not be plotted. The last argument says why, counted in rows, naming only the reasons that apply, joined with , ; each is one of

warn_map_rows_why_empty      %d with no place in the row
warn_map_rows_why_refused    %d whose place is a bare number, not a place name
warn_map_rows_why_not_found  %d whose place was not found
warn_map_rows_why_later      %d whose place has not been looked up yet

in that order, rendered through the same i18n object as the message. Not an error: the map renders with the rows that could be resolved.

warn_map_geocode_capped

Geocoded %d of the %d places that had not been looked up before; ...

The max cap, or online_max, prevented all pending values from being looked up in this request. The un-looked-up values are in not_attempted and are not stored in the cache, so the next request picks them up.

warn_map_geocoder_unavailable_data

The geocoder has no OpenAddresses data configured (OPENADDR_HOME), ...

The geocoder was loaded successfully but OPENADDR_HOME is not set, openaddr was not passed to new(), and Nominatim is not in the chain. Resolution will succeed for broad place names (cities, countries) but will fail for street-level addresses. Set the environment variable or pass openaddr => $path to new().

warn_map_geocoder_local_only

Places the local data does not know were not looked up online, which needs Geo::Coder::OSM 0.03 and LWP::UserAgent::Throttled 0.14 or later

Online lookups were wanted, but one of the two modules is missing or too old, and some places were not found. Install them, or pass online => 0 to say the local data is all that is wanted.

warn_map_geocoder_online_failed

OpenStreetMap did not answer (%s), so %d places the local data does not know are left to try again later

Nominatim answered with an error (the %s is its status line, such as 429 Too Many Requests) or not at all. The places are in not_attempted, uncached, and the next request asks again.

FORMAL SPECIFICATION

The following sections use Z calculus notation to describe the exact state transitions and invariants of each public operation. Notation: ∈ (member of), ¬in; (not member of), ⊆ (subset), ∪ (union), ∩ (intersection), ∧ (and), ∨ (or), ¬ (not), ∀ (for all), ∃ (there exists), → (maps to), ≤ (less than or equal), ≠ (not equal).

FORMAL SPECIFICATION

State transitions and invariants for each operation are described below.

_resolve -- value classification and batch geocoding

Let V be the multiset of input values.

Define the following disjoint sets that partition distinct(V):

SKIPPED          = { v | v = undef, ref(v), or len(v) = 0,
                         or geocoder.valid_location(v) = false }
CACHED_RESOLVED  = { v | v in dom(CACHE) and _has_position(CACHE(v)) }
CACHED_UNRESOLVED = { v | v in dom(CACHE) and not _has_position(CACHE(v)) }
NOT_ATTEMPTED    = { v | geocoder unavailable } (excludes SKIPPED and CACHED)
PENDING          = distinct(V) \ (SKIPPED ∪ CACHED_RESOLVED ∪ CACHED_UNRESOLVED
                                  ∪ NOT_ATTEMPTED)

Partition invariant:

|SKIPPED| + |CACHED_RESOLVED| + |CACHED_UNRESOLVED|
    + |NOT_ATTEMPTED| + |PENDING| = |distinct(V)|

Cap application (MAX = the max option, default 500):

ATTEMPT        = PENDING[0 .. min(MAX, |PENDING|) - 1]
CAPPED         = |PENDING| > MAX
NOT_ATTEMPTED' = NOT_ATTEMPTED ∪ PENDING[MAX .. |PENDING| - 1]   (when CAPPED)

Deferral (lookup => 0):

R.pending      = ATTEMPT
NOT_ATTEMPTED' = NOT_ATTEMPTED' ∪ ATTEMPT
ATTEMPT        = ∅                 (so CACHE' = CACHE, and nothing is asked)
CAPPED         = false

Otherwise R.pending = ∅.

Geocoding step -- for each v in ATTEMPT:

BATCH = { v => chain.geocode(v) | v in ATTEMPT }    (each through the list's
        cache; an injected geocoder: geocode_many(ATTEMPT, on_error => 'undef');
        a v whose lookup threw and found nothing moves to NOT_ATTEMPTED)

for v in ATTEMPT:
    (lat, lon) = coordinates(BATCH(v))
    if defined(lat) and defined(lon):
        CACHE'(v)  = [lat, lon]
        POINTS'(v) = { lat, lon, backend? }
    else:
        CACHE'(v)  = 0
        UNRESOLVED_NEW += {v}

Count invariants (always hold):

skipped + cached + attempted + not_attempted = distinct(V)
resolved + unresolved = cached + attempted
resolved  = |CACHED_RESOLVED| + |ATTEMPT ∩ POINTS'|
unresolved = |CACHED_UNRESOLVED| + |ATTEMPT \ POINTS'|
lookups   <= attempted

Cache monotonicity:

dom(CACHE') ⊇ dom(CACHE)
For all v in dom(CACHE): CACHE'(v) = CACHE(v)  (existing entries not modified)

resolve_column -- row augmentation

Given the output R of _resolve applied to the column's cell sequence CELLS:

row_count  : Value -> N   where row_count(v) = |{ i | CELLS[i] = v }|
first_row  : Value -> N   where first_row(v)  = min{ i | CELLS[i] = v }

For each v in dom(R.points):
    R.points(v).rows      = row_count(v)
    R.points(v).first_row = first_row(v)

R.markers = [ { value: v, ...R.points(v) }
              | v <- sort_by(first_row, dom(R.points)) ]

R.counts.rows     = |CELLS|
R.counts.plotted  = sum{ row_count(v) | v in dom(R.points) }
R.counts.unusable = R.counts.rows - R.counts.plotted

cache_from_models -- safe extraction

cache_from_models(models, spec)
    = if ref(models) = HASH and ref(models.geo.spec) = HASH
                            and ref(models.geo.spec.places) = HASH
        then copy(models.geo.spec.places)
        else {}

The result is a shallow copy: the caller cannot modify the model store by modifying the returned hashref, but values inside the cache (the [lat, lon] arrayrefs) are shared references.

models_with_cache -- safe merge

models_with_cache(models, spec, cache, backend?, built?)
    = { ...models,
        geo: { ...models.geo,
               spec: { ...models.geo.spec,
                        places:  copy(cache),
                        backend: backend (if defined),
                        built:   built   (if defined) } } }

The caller's models hashref is never modified; a new top-level hashref is returned.

STATE DIAGRAM

The object is in one of three states, according to whether it has a geocoder yet. A geocoder given to new is one straight away; otherwise one is looked for on the first call that needs it.

  new()
    |
    +--[bad arguments, or a geocoder lacking
    |   valid_location or geocode_many]-----> croaks; no object
    |
    +--[geocoder => $g]---------------------> READY
    |
    v
+-------------------+
|    CONSTRUCTED    |  Geocoder not yet lazy-loaded.
|  _geocoder not    |  resolve_*() and geocoder_available()
|  yet called       |  may trigger a load on first call.
+-------------------+
     |          |
first call,     first call,
module OK       module absent, too old or will not
     |          load, or its new() answers nothing
     v          v
+---------+  +------------+
|  READY  |  | UNAVAILABLE|
|         |  |            |
| Geocoder|  | No lookups |
| loaded  |  | possible.  |
| and     |  | Cache still|
| working.|  | served.    |
+---------+  +------------+
     |          |
resolve_*()  resolve_*()
geocodes +   cache only;
cache        not_attempted
     |          |
     v          v
+---------+  +------------+
|  READY  |  | UNAVAILABLE|
| (no     |  | (no        |
| change) |  | change)    |
+---------+  +------------+

Once the object is READY or UNAVAILABLE that does not change: neither a module that appears nor one that goes away moves it, and the module is looked for once.

There are two ways for a call to croak and leave the state as it was. Arguments a resolve_*() call refuses are refused before any load is attempted. A module that loads and whose new() dies sends that exception to the caller and remembers nothing, so the object stays CONSTRUCTED and the next call tries again.

Resolution itself is stateless: all state is in the result hashref and the cache the caller manages.

Valid state transitions summary:

[START]          -> CONSTRUCTED      (new() with no geocoder)
[START]          -> READY            (new() given a geocoder)
[START]          -> ERROR            (new() refuses its arguments; croaks, no object)
CONSTRUCTED      -> READY            (first call, module OK)
CONSTRUCTED      -> UNAVAILABLE      (first call, no module that can be used)
CONSTRUCTED      -> CONSTRUCTED      (arguments refused, or the module's new() dies; croaks)
READY            -> READY            (resolve_*(), or arguments refused)
UNAVAILABLE      -> UNAVAILABLE      (resolve_*(), or arguments refused)

Value-level state transitions within a single resolve_* call. Each distinct value takes the first of these that applies, from the top. CACHE is the cache in the result, a copy: the cache the caller passed in is never modified.

RECEIVED
   |
   +--[undef / ref / empty]-----------> SKIPPED
   |
   +--[in cache, has position]---------> CACHED_RESOLVED --> POINTS
   |
   +--[in cache, no position (= 0)]----> CACHED_UNRESOLVED --> UNRESOLVED
   |
   +--[geocoder unavailable]-----------> NOT_ATTEMPTED
   |
   +--[valid_location() throws]--------> EXCEPTION
   |
   +--[valid_location() = false]-------> SKIPPED
   |
   +--[within cap]--------------------> ATTEMPT
   |                                        |
   |                                        +--[geocoder returns position]--> RESOLVED_NEW --> POINTS, CACHE==[lat,lon]
   |                                        |
   |                                        +--[geocoder returns no position]--> UNRESOLVED_NEW, CACHE==0
   |                                        |
   |                                        +--[geocoder throws]--> EXCEPTION
   |
   +--[beyond cap]-------------------> NOT_ATTEMPTED

EXCEPTION is the whole request failing, on purpose: it goes to the caller, who has no result and a cache exactly as it was, so nothing can be half-written.

AUTHOR

Nigel Horne <njh@nigelhorne.com>

LICENCE AND COPYRIGHT

Usage is subject to the GPL2 licence terms. If you use it, please let me know.

To install Database::BI, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Database::BI

CPAN shell

perl -MCPAN -e shell
install Database::BI

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

DESCRIPTION

The cache

Without the geocoder

CONSTRUCTOR

new

API SPECIFICATION

INPUT

OUTPUT

SUBROUTINES/METHODS

resolve_column

API SPECIFICATION

INPUT

OUTPUT

MESSAGES

EXAMPLE

resolve_values

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

resolve_column

geocoder_available

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

file_cache

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

place_rows

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

cache_from_models

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

models_with_cache

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

shared_cache_from_models

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

models_with_shared_cache

API SPECIFICATION

INPUT

OUTPUT

EXAMPLE

COMMON PITFALLS

ENCODING

DIAGNOSTICS

FORMAL SPECIFICATION

FORMAL SPECIFICATION

_resolve -- value classification and batch geocoding

resolve_column -- row augmentation

cache_from_models -- safe extraction

models_with_cache -- safe merge

STATE DIAGRAM

AUTHOR

LICENCE AND COPYRIGHT

Module Install Instructions

Keyboard Shortcuts