/ 
Database-BI-0.001.0
/ Database::BI::Model::DataSource

CONSTRUCTOR

new

Creates and returns a new Database::BI::Model::DataSource instance.

API SPECIFICATION

INPUT

{
    directory => 'string',           # required; path to the data directory
    table     => 'string',           # required; bare table/file name (no extension)
    i18n      => { type => 'object', optional => 1, can => 'maketext' } # must implement maketext($key, @args)
}

Accepts a flat key/value list, a hashref, or positional arguments via Params::Get.

DOMAIN CONSTRAINTS

directory

Must satisfy -d $directory (must exist and be a directory). An empty string, a non-existent path, or the path to a regular file all produce error_directory_missing.

Valid partition:   any existing directory path
Invalid partition: non-existent path, regular file path, empty string ""
table

Must match TABLE_NAME_RE = \A[A-Za-z_][A-Za-z0-9_]*\z. The first character must be a letter (A-Z, a-z) or underscore; subsequent characters may also be digits.

Valid partition:   "sales", "_tmp", "report_2024" (letter/underscore start)
Invalid partition: "1sales" (digit-start), "my.data" (dot),
                   "my-data" (hyphen), "" (empty string)
Boundary values:   "a" (length-1 letter, valid), "_" (length-1 underscore,
                   valid), "1" (length-1 digit, croaks error_table_name_invalid)

OUTPUT

Returns $self (a blessed hashref). Croaks on invalid arguments.

MESSAGES

error_directory_required    -- "directory" argument was not supplied
error_table_required        -- "table" argument was not supplied
error_directory_missing     -- supplied directory does not exist / is unreadable
error_table_name_invalid    -- table name fails the safe-identifier check
error_backend_init          -- Database::Abstraction subclass could not be instantiated

FORMAL SPECIFICATION

new == [directory : PATH; table : NAME; i18n? : I18N_OBJECT]
       pre  (directory in dom FILE_SYSTEM /\ is_dir directory)
            /\ table =~ TABLE_NAME_RE
       post result.class = DataSource
            /\ result._db.class = Database::Abstraction

ACCESSORS

table_name

Returns the (lowercased) table name this instance was opened against.

API SPECIFICATION

INPUT

None.

OUTPUT

Returns a SCALAR string.

MESSAGES

None.

FORMAL SPECIFICATION

table_name == lambda self . self._table

columns

Returns an arrayref of column names in file order, or undef when the backend does not expose a fixed column order (e.g. SQLite, XML).

id_column

Returns the name of the column used as the primary key / slurp-filter anchor. Returns undef for URL/HTML-table backends (no primary-key concept applies).

source_url

Returns the source URL for URL/HTML-table-backed instances, or undef for file-backed instances.

METHODS

fetch_all

Returns every record in the table as an arrayref of hashrefs.

API SPECIFICATION

INPUT

None. Filtering is performed at the controller layer (Dashboard::_apply_filter_spec) after fetch_all returns, not inside DataSource.

OUTPUT

ARRAYREF of HASHREF   # one hashref per row, keys are column names
                      # returns [] when the table exists but is empty

Croaks if the backend raises an exception. Carps (non-fatal) when the result set is empty so the caller can distinguish "open succeeded, no rows" from a silent failure.

MESSAGES

error_fetch_failed     -- backend threw an exception during retrieval
warn_empty_result      -- query succeeded but returned zero records
warn_data_normalised   -- backend returned a hashref; converted to arrayref

FORMAL SPECIFICATION

fetch_all == lambda self .
  let rows = self._db.selectall_hashref() in
  pre  self._db /= undef
  post result : seq HASHREF
       /\ #result >= 0

NAME

Database::BI::Model::DataSource - Table-agnostic adapter around Database::Abstraction

VERSION

This document describes Database::BI::Model::DataSource version 0.01.

SYNOPSIS

Read all rows from a CSV file:

use Database::BI::Model::DataSource;

my $source = Database::BI::Model::DataSource->new(
    directory => '/path/to/data',
    table     => 'sales',           # looks for data/sales.csv, .psv, .sql, .xml, etc.
);

my $records = $source->fetch_all;   # arrayref of hashrefs -- one hashref per row

for my $row (@{$records}) {
    printf "Product: %s, Amount: %s\n", $row->{product}, $row->{amount};
}

Get column names in the original file order (CSV/PSV only):

my $cols = $source->columns;        # returns undef for SQLite and XML
if ($cols) {
    print join(', ', @{$cols}), "\n";
}

Find out which column is the primary key:

print "Primary key column: ", $source->id_column, "\n";

Open a SQLite file (.sql extension):

my $source = Database::BI::Model::DataSource->new(
    directory => '/var/data',
    table     => 'inventory',       # looks for /var/data/inventory.sql
);

Open a pipe-separated file (.psv extension):

my $source = Database::BI::Model::DataSource->new(
    directory => '/var/data',
    table     => 'products',        # looks for /var/data/products.psv
);

Use a custom i18n object to translate error messages:

# The i18n object must have a maketext($key, @args) method.
my $source = Database::BI::Model::DataSource->new(
    directory => '/path/to/data',
    table     => 'sales',
    i18n      => My::I18N::Handle->new,
);

Handle errors gracefully:

my $source = eval {
    Database::BI::Model::DataSource->new(
        directory => $dir,
        table     => $table,
    );
};
if ($@) {
    carp "Could not open table: $@";
    # $@ contains a translated message from %MESSAGES
}

my $records = eval { $source->fetch_all };
if ($@) {
    carp "Could not read records: $@";
}

DESCRIPTION

Database::BI::Model::DataSource is a thin, table-agnostic adapter that wraps Database::Abstraction and exposes three accessors (fetch_all, columns, id_column) used by the controller.

Database::Abstraction is a read-only ORM that discovers data files (CSV, PSV, SQLite, XML, etc.) automatically from a directory based on the calling class name. DataSource generates an ephemeral subclass at construction time so callers never interact with Database::Abstraction directly. To swap the backend for Database::Join in Phase 2, only the open_table helper in Database::BI needs to change; the controller and DataSource are untouched.

_detect_file_info peeks at the first header line of CSV/PSV files to extract the correct separator character, the primary-key column name, and the full ordered column list. Without this, two silent Database::Abstraction defaults corrupt every result: sep_char defaults to '!' (turning a comma-separated file into a single-field table) and id defaults to 'entry' (causing every row to be discarded when no entry column exists).

Result filtering (eq, contains, gt, etc.) is performed at the controller layer by Dashboard::_apply_filter_spec after fetch_all returns. DataSource itself is filter-unaware.

All user-visible strings and exception messages are keyed through the %MESSAGES dictionary and routed via _msg(), making every diagnostic replaceable by an i18n object at instantiation time.

COMMON PITFALLS

These are the most common mistakes when using DataSource.

SQLite databases must use .sql as their file extension

Database::Abstraction looks for SQLite databases with the .sql extension. It does not recognise .sqlite, .sqlite3, or .db3. If you have a file called inventory.sqlite, rename it to inventory.sql before passing it to DataSource.

The table name is always lowercased

The constructor lowercases the table argument before using it. Passing table => 'Sales' and table => 'sales' both look for sales.csv (or sales.sql, etc.). The matching is case-insensitive on the table name but case-sensitive on the directory path.

CSV files with the wrong separator appear as one giant field per row

Database::Abstraction defaults to ! (exclamation mark) as its field separator. A standard comma-separated CSV file will look like one big field per row (for example, 1,Widget A,North,100) because the library never sees the commas as separators. DataSource fixes this automatically by reading the first line of the file and detecting the actual separator. If you bypass DataSource and call Database::Abstraction directly, you must pass sep_char => ',' yourself.

A table with no "entry" column returns zero rows (without DataSource)

Database::Abstraction uses entry as its default primary-key column name. When slurping a CSV, it discards any row where $row->{entry} is undefined. Because most CSV files do not have an entry column, all rows are silently discarded. DataSource prevents this by reading the actual first column name from the CSV header and passing it as id, and also by setting no_entry => 1 so all rows are kept as an ordered array.

columns() returns undef for SQLite and XML files

columns() only returns an arrayref for file formats where the header order is visible before data is read (CSV and PSV). For SQLite and XML files it returns undef. Always check: if ($source->columns) { ... }. The controller falls back to putting id_column first and then sorting the rest alphabetically when columns() is undef.

XML elements named "name", "id", or "key" cause parse failures

XML::Simple (used by Database::Abstraction for XML files) automatically turns a child element called name, id, or key into a hash key instead of keeping it as an array element. This breaks the expected data structure. Use different element names in your XML: for example, <sku>, <label>, or <code> instead of <id> and <name>.

<!-- WRONG: these element names trigger XMLin key-folding -->
<items>
  <item><id>1</id><name>Widget</name></item>
</items>

<!-- CORRECT: use neutral element names -->
<items>
  <item><sku>1</sku><label>Widget</label></item>
</items>
fetch_all returns an empty arrayref, not undef, for an empty table

When a table exists but contains no data rows, fetch_all returns [] (an empty arrayref), not undef. Check with scalar @{$records}, not with defined $records or $records.

LIMITATIONS

  • Only read operations are supported. Write-back is not in scope.

  • One DataSource instance corresponds to exactly one table. Multi-table left joins are composed at the controller layer by Dashboard::_left_join; Database::Join (Phase 2) is not yet in use.

  • The ephemeral backend class is generated into a package namespace (Database::BI::_DB::*) that persists for the lifetime of the process. Instantiating two DataSource objects for the same table name reuses the same ephemeral class.

  • The i18n object, if supplied, must implement maketext($key, @args) compatible with Locale::Maketext.

CONFIGURATION AND ENVIRONMENT

No environment variables are read. All configuration is passed through the constructor.

DEPENDENCIES

Carp, Readonly, Scalar::Util, Params::Validate::Strict, Params::Get, Database::Abstraction.

INCOMPATIBILITIES

None known.

BUGS AND LIMITATIONS

Please report bugs via https://github.com/nigelhorne/Database-BI/issues.

AUTHOR

Nigel Horne <njh@nigelhorne.com>

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the GPL2 licence terms. If you use it, please let me know.