Revision history for Database-BI
0.006.0 Tue Sep 15 08:03:20 PM EDT 2026
[ Enhancements ]
- Added a 'home' link on the table display page
- XLSX files are now supported directly via DBD::Excel (each worksheet
is a separate table). The upload endpoint, filesystem browser, and
drag-and-drop drop zone all accept .xlsx files. DBD::Excel added to
TEST_REQUIRES.
- "Combine data" now allows a file to be drag and dropped
- Multi-table left joins are now delegated to Database::Join (>= 0.003.0)
instead of the bespoke _left_join helper. Column collision handling
(prefixing right-table columns with "label.col" when a name clashes) is
provided by Database::Join's new collision_prefix parameter.
DataSource now exposes selectall_arrayref() and an improved columns()
fallback so that DataSource objects can be passed directly to
Database::Join as component databases.
- Database::Join added to PREREQ_PM (runtime dependency).
- DataSource::new now silently sanitizes table names derived from filenames:
characters that are illegal in SQL identifiers (hyphens, dots, spaces,
etc.) are replaced with underscores, and a digit-leading stem is prefixed
with '_'. The original filename stem is kept as the dbname so
Database::Abstraction still locates the correct file on disk. This means
files like "Transactions-2026-09-08.csv" and "1data.csv" can now be
opened via /open without error. Only path-separator characters ('/', '\',
NUL) and the empty string continue to croak error_table_name_invalid.
- A "Totals" checkbox in the toolbar appends a summary row to the bottom of
any data table. When checked, every column whose values are all numeric
(using the same parseAccounting logic as the sort and graph features, so
accounting-notation negatives are handled correctly) shows its sum; date
columns (YYYY-MM-DD, M/D/YYYY) are excluded from totalling even though
parseFloat can extract a number from them. Non-numeric columns are left
blank. The checkbox state is persisted in localStorage alongside column
order and sort direction, so the totals row reappears automatically on
page reload. The summary row follows column drag-to-reorder automatically
because moveCol() operates on all table rows including tfoot.
- CSV and PSV files with no header row are now handled automatically.
When every value in the first line fails the safe-identifier check,
_detect_file_info tests whether any value looks like a date (YYYY-MM-DD,
M/D/YYYY) or a signed/accounting-notation number. If so, the file is
treated as header-less: all rows (including the first) are read directly
and column names are synthesised from the inferred data types — date
values become "Date", numeric/currency values become "Amount", and free
text becomes "Description". Duplicate types are disambiguated with a
numeric suffix (Date2, Amount2, ...). This allows bank and accounting
exports such as "Transactions-current.csv" to open without error or any
manual renaming of headers.
- CHI in-process caching is now active for all data sources. URL-backed
tables (fetched via LWP from a remote web page) are cached for 15 minutes
using a TTL-based key, so repeated visits to the same /import URL avoid
a network round-trip on every page load. File-backed tables are cached
using a key that encodes the file's mtime, so a changed file naturally
produces a cache miss without any explicit invalidation — the stale entry
is orphaned and reclaimed when the process restarts or the CHI Memory
driver's GC runs. The cache TTL and driver can be overridden in
database_bi.conf via the "cache" key: { driver => 'File', ttl_url => '5 min' }.
CHI added to PREREQ_PM and cpanfile.
- DataSource::columns() now returns undef for URL-backed tables instead of
delegating to Database::Abstraction's columns(). D::A fetches URL data
lazily (in _open(), not in new()), so calling _db->columns() after a CHI
cache hit (which bypasses _open()) would trigger an unexpected live network
request — failing with a non-200 error even when valid cached data was
available. Returning undef lets _get_columns derive column names from the
cached data records, which is correct since URL/HTML tables have no
canonical column order anyway.
- Rows and columns can now be selected and deleted in the virtual table view.
A checkbox column is injected at the left edge of every data table;
clicking a row's checkbox selects it (turning the row pink) and Ctrl+click
on any column header selects the column (highlighting all its cells in red).
A "Delete selected" button appears in the toolbar as soon as any rows or
columns are selected; clicking it removes them from the in-browser view
without touching the underlying file. Row selection supports a
select-all/deselect-all checkbox in the header. The record count in the
toolbar is updated after deletion. Sorting, drag-to-reorder, and the
Totals row all continue to work correctly after deletions.
- Line graphs now animate on initial page load (requires HTML::D3 >= 0.13).
The line draws left-to-right via a stroke-dashoffset transition (1200 ms,
easeLinear); data-point circles fade in after the line finishes (300 ms
fade, 1200 ms delay). Zoom and reset redraws are not animated — a
per-page initialDrawDone flag ensures the effect fires only once.
When the viewer's OS has prefers-reduced-motion enabled the animation
is skipped entirely and the chart appears instantly.
HTML::D3 minimum version bumped to 0.13 in PREREQ_PM and cpanfile.
[ Bug Fixes ]
- When attempting to drag and drop a file larger than MAX_UPLOAD_BYTES,
a better error message is displayed which stays on the screen for
3 seconds
- CSV and PSV files whose first line is blank (e.g. a file containing only
a single newline character) are now treated as empty files rather than
croaking "no column with a safe identifier name". _detect_file_info now
returns the _file_is_empty sentinel when the parsed first line yields
zero column names, causing _init_backend to skip Database::Abstraction
entirely and fetch_all to return []. The /open endpoint renders the
standard empty-table view ("No records found") instead of an error page.
Tests added to t/edge_cases.t.
- Files whose names contain spaces (e.g. "transactions for Nigel.xlsx")
now open correctly. Database::Abstraction validates its dbname argument
as a SQL identifier and rejected names with spaces with an "unsafe dbname"
error at fetch time. DataSource::_init_backend now creates a temporary
directory with a symlink using the sanitized name whenever the raw
filename stem contains illegal SQL characters, so D::A sees only the
safe name. Regression test added as Transaction 27.
- The "Line graph..." toolbar button was silently broken: clicking it did
nothing. buildYSelect() referenced SEL from the table-management IIFE
where it was declared, but the graph-panel IIFE lives in a separate
<script> block and cannot access that scope. The ReferenceError thrown
during initGraphBtn() prevented the click listener from ever being
registered. Fixed by removing the cross-IIFE reference entirely:
buildYSelect() now uses r.cells[idx] instead of r.cells[idx + SEL],
because Array.from(tHead.rows[0].cells) already includes the injected
sel-th at position 0, so both the header and data arrays are shifted
identically and no additional offset is needed. A regression test in
Transaction 22 verifies that buildYSelect does not reintroduce
the idx+SEL double-offset.
- The Y-axis dropdown in the "Line graph..." panel was listing non-numeric
columns (e.g. Description) as valid Y-axis choices, and omitting the
correct numeric ones (e.g. Balance). Root cause: buildYSelect() used
r.cells[idx + SEL] where idx came from headers.forEach() over
tHead.rows[0].cells. Because injectSelCol shifts both thead (sel-th)
and tbody (checkbox) by one position, the header and data arrays are
already aligned; adding SEL read one cell to the right of the header
under test, causing every column's numeric check to be evaluated against
the data of the column to its right. Fix: changed to r.cells[idx].
- Spreadsheet::ParseXLSX added to PREREQ_PM. DBD::Excel 0.07 only
handles .xls (its source skips .xlsx files entirely); DataSource now
reads .xlsx directly via Spreadsheet::ParseXLSX in _detect_file_info,
bypassing D::A the same way headerless CSV files do.
Excel::Writer::XLSX added to TEST_REQUIRES for fixture creation.
Transaction 28 covers XLSX open lifecycle including spaced filenames.
- Synthesised column headers for header-less CSV/PSV files are now
capitalised (Date, Amount, Description) so they display correctly as
table headers in the browser.
- Berkeley DB files (.db extension) can now be opened via /view/:table,
/open, and the filesystem browser. Database::Abstraction detects the
BerkeleyDB magic bytes automatically and opens the file via DB_File
(a Perl core module). No code change was required in Database::BI;
the existing @SUPPORTED_EXT and _detect_file_info already handled .db
files correctly. Transaction 29 confirms the end-to-end lifecycle.
0.005.2 Fri Sep 11 06:08:10 AM EDT 2026
[ Bug Fixes ]
- Fixed SIGABRT crash (DBI XS assertion failure) in t/edge_cases.t on
DEBUGGING-enabled perls (observed: perl 5.44.0 + DBI 1.651 + DBD::CSV
0.64). When a 0-byte CSV file was opened, _detect_file_info returned
the generic "not CSV" sentinel ({}), causing _init_backend to create a
Database::Abstraction object without the max_slurp_size guard. D::A
fell back to the DBD::CSV path, which issued "SELECT * FROM <table>" on
an empty file; DBD::CSV's EOF error left DBI's internal Errstr SV in a
shape that violated an XS assertion, crashing the process during global
destruction after all test assertions had already passed (so the TAP
plan was never written and the harness reported FAIL).
Fix: _detect_file_info now returns { _file_is_empty => 1, file_size => 0 }
for 0-byte files. _init_backend detects this sentinel and short-circuits
before creating D::A; fetch_all returns [] immediately. DBI is never
contacted and no crash occurs.
References: https://www.cpantesters.org/cpan/report/117b2a76-a4e5-11f1-9192-ffecfaad6fbc
- Reference lines (Min / Avg / Max) on the line graph now recalculate
correctly after a brush-to-zoom or Reset zoom interaction. Two bugs
were responsible:
* The previous implementation tried to hook into the zoom by listening
for a D3 brush "end" DOM event, but D3 fires brush callbacks through
its own internal dispatch, not through DOM events. The listener
never fired. The fix patches the global `redraw` function (a
function declaration, not a `const`) and recalculates the reference
values (min / avg / max) from the data currently being drawn
(`newData`) rather than the full-dataset values stored at page load.
This ensures the lines always fall within the visible y-domain even
after zooming into a subset of the data.
* `drawRefLines()` was called synchronously during the 500 ms D3
transition. Although `y.domain()` is updated before the transition
begins, the y-axis visually animates to the new domain over 500 ms;
appending lines mid-transition produced a brief positional mismatch.
The call is now deferred with `setTimeout(drawRefLines, ms + 20)`.
- Requires HTML::D3 >= 0.11 for the companion fix: the zoomable line
chart y-domain now uses `Math.min(0, d3.min(...))` as its lower bound
instead of hard-coding 0. This ensures data points and reference
lines with negative Y values (e.g. accounting-notation amounts) are
plotted within the visible chart area rather than appearing off-screen
below the x-axis.
0.005.1 Sun Aug 30 08:29:03 PM EDT 2026
[ Bug Fixes ]
- Fixed compatibility with Database::Abstraction >= 0.41. D::A 0.41 changed
dbname to fall back to the class-name suffix rather than the table parameter,
so a dynamically-generated package like Database::BI::_DB::Orders was used
as the filename stem (Orders.csv) instead of the lowercase table name
(orders.csv), causing "Can't find a file" errors on case-sensitive Linux
filesystems. DataSource::_init_backend now passes dbname => $table
explicitly so the filename stem is always correct regardless of D::A version.
Minimum Database::Abstraction version bumped to 0.41.
- Columns containing accounting-style negative amounts - values written as
(1,234.56) rather than -1,234.56 - are now recognised as numeric and
handled correctly throughout the application:
* Column sort: clicking an Amount header now sorts correctly, with
parenthesised values treated as negative numbers rather than being
sorted as text or treated as positive values.
* Y-axis selector: columns whose values use accounting notation are now
included in the "numeric columns" list for line graphs, so Amount (and
similar columns) appear in the Y-axis dropdown.
* Line graph: Y values in accounting notation are plotted at the correct
negative position on the chart rather than at their absolute value.
The fix adds a parseAccounting() helper in the browser (shared by sort
and graph detection) and updates the server-side graph_view Y-strip logic
to detect the leading '(' before stripping non-numeric characters.
0.005.0 Sat Aug 29 02:22:00 PM EDT 2026
[ Enhancements ]
- The "Line graph..." toolbar button is now greyed out (disabled) when the
current view has no numeric column that can serve as the Y axis.
Hovering over the disabled button shows the tooltip "Line graphs need a
numeric column". The check runs once at page load using the same
isNumericVal / isDateVal logic as the Y-axis column selector, so date
columns and fully-text columns both correctly suppress the button.
- When a view has exactly one numeric column the Y-axis dropdown is
replaced by a read-only "auto-selected" label showing the column name,
so the user only needs to choose the X axis before clicking Plot.
- Upload subdirectories older than 24 hours are now evicted automatically
on every server startup. Recent uploads (from the current or previous
session within the last 24 hours) are preserved. The explicit
"Clear upload cache" button (POST /uploads/clear) continues to remove
all entries immediately regardless of age.
- Graph hover tooltips now show all columns for the data point, not just
the X and Y values. Mousing over a dot displays every field from that
row (excluding the axes columns) as additional key/value lines below the
label and value. Requires HTML::D3 >= 0.09.
- Line graph now supports brush-to-zoom: drag across a range on the X axis
to zoom in; a "Reset zoom" button returns to the full view. Uses
HTML::D3's render_zoomable_line_chart_snippet.
- Line graph toolbar now shows the number of plotted data points
(e.g. "Plotting 6 points") so users know immediately how much data
is in view, especially after filtering.
- GET /export now accepts format=json to download the current view as a
JSON array of objects. CSV and SQLite remain available as before.
- GET /api/columns now accepts a ?spec= parameter in the unified
"table:name" or "path:/abs/path" format used by /join and /graph,
so JavaScript can pass the l= pipeline spec directly without parsing it.
[ Bug Fixes ]
- Graph page correctly returns HTTP 200 with a "No plottable data" message
when the chosen Y column contains no numeric values (e.g. a text column),
rather than crashing or producing an empty chart.
- The first X-axis label on the line graph is no longer cropped. D3
rotates labels -45 degrees with text-anchor:end, so the leftmost
label extends to a negative SVG x-coordinate. The fix widens the
SVG via a viewBox that starts at x=-60: the formerly-negative region
becomes part of the SVG viewport, so no CSS overflow trick is needed.
- t/filter.t subtest "import_url: non-numeric table_index is sanitized to 0"
now passes on GitHub Actions. Database::Abstraction fetches URLs via
LWP::UserAgent::Cached (not plain LWP::UserAgent); the test was mocking
LWP::UserAgent::get but that mock was bypassed by Cached's override of
simple_request. The mock now targets LWP::UserAgent::simple_request
(the base-class method that Cached delegates to after its cache check),
which intercepts the call regardless of which LWP subclass D::A uses.
LWP::UserAgent::Cached is also added to PREREQ_PM and cpanfile so CI
installs it; previously it was absent and the feature silently failed
on fresh environments.
0.004.0 Thu Aug 27 09:36:22 PM EDT 2026
[ Enhancements ]
- Date columns now sort chronologically when clicking the column header.
The sort routine detects slash-separated dates (M/D/YYYY and D/M/YYYY)
and ISO 8601 dates (YYYY-MM-DD) and converts them to a YYYYMMDD integer
key before comparing, so 1/5/2017 correctly sorts after 12/27/2016.
For slash dates the format is auto-detected by scanning for a component
that exceeds 12: if the second component ever exceeds 12 the column is
month-first (M/D/YYYY); if the first component exceeds 12 it is
day-first (D/M/YYYY); ambiguous columns default to month-first.
- Add line graph feature: a "Line graph..." button on any data view opens
a column-picker panel where the user selects X and Y axes. Clicking
"Plot" navigates to GET /graph, which opens the same data pipeline as
the current view (joins, combines, filters, dedup) and renders an
interactive D3.js v7 line chart with hover tooltips via HTML::D3.
The graph page includes a toolbar injected into the HTML::D3 output
with a "Back to table" link (returning to the exact URL the user was
on) and "Export SVG" / "Export PNG" buttons. SVG is serialized with
XMLSerializer; PNG is rasterized via a Canvas element.
Non-numeric Y values (currency symbols, commas, etc.) are stripped and
the numeric portion is used; rows where Y is still non-numeric after
stripping are silently skipped.
[ Bug Fixes ]
- Drag-and-dropped files whose original filesystem path cannot be
determined are no longer added to the "recently opened" list.
Previously the temporary .uploads/<tempdir>/ path was recorded as a
fallback, producing a useless entry that pointed inside the app's
upload cache. Now, when no file:// URI is available from the
DataTransfer text/uri-list data (e.g. Firefox, email attachments),
the entry is omitted entirely. Files whose original path IS
recoverable continue to appear in the list under that path.
[ Test Improvements ]
- Added Transaction 20 subtest covering graph UI polish and date-sort JS:
verifies biDateKey helper and monthFirst detection are present, Plot
precedes Cancel in the graph panel DOM, chart centering CSS is injected,
and the back link carries the correct colour/size/weight.
0.003.2 Wed Aug 26 09:24:45 PM EDT 2026
[ Bug Fixes ]
- Large CSV files (> 16 KB) with column names containing spaces — such as
bank-export files with an "Account Number" column — now render correctly.
Files exceeding Database::Abstraction's default 16 KB slurp threshold
previously fell through to the DBD::CSV SQL path, which sanitizes column
names (lowercases them and replaces spaces with underscores, so "Account
Number" becomes "account_number"). The template then tried to access the
original name in a restricted hash that only permitted the sanitized key,
producing "Attempt to access disallowed key 'Account Number' in a
restricted hash". DataSource::_detect_file_info now returns the file
size, and _init_backend passes it as max_slurp_size to
Database::Abstraction, ensuring the Text::xSV::Slurp path is always
taken for CSV/PSV files so original column names are preserved regardless
of file size.
[ Test Improvements ]
- Added path-K and path-K2 subtests in t/path.t asserting that
_detect_file_info now returns file_size in its result hashref for
CSV/PSV files and omits it for the empty-hashref (non-CSV) case.
- Added end-to-end regression subtest BV-large-spaced-header in t/domain.t:
creates a > 16 KB CSV with "Account Number" as a column header, opens
it via /open, and asserts all rows are visible and no restricted-hash
crash occurs.
0.003.1 Wed Aug 26 04:54:25 PM EDT 2026
[ Enhancements ]
- Allow files to be removed from the recently opened and recently saved
lists
[ Bug Fixes ]
- Drag-and-dropped files now appear in "Recently opened" under their
original filesystem location, not the temporary .uploads/ copy. The
drop handler now extracts the file:// URI from the DataTransfer
text/uri-list item (provided by desktop file managers such as Nautilus
and Finder) and stashes it in sessionStorage before starting the
upload. The dashboard page then reads that stash and records the real
path. If no file:// URI is available (browser drag, email attachment,
etc.), the entry is omitted entirely so a useless temp path never
- Dragging and dropping the same file more than once no longer creates
duplicate entries in the "Recently opened" section on the home page.
Each upload creates a new random .uploads/<tempdir>/ subdirectory, so
the href changed on every drop even for the same file, bypassing the
existing href-based deduplication. The dedup key is now normalised:
upload entries are keyed by display name ("upload:<filename>"); all
other entries (browser-opened files, URL imports) continue to be keyed
by their full href so that two different files with the same name from
different directories can both appear in the list.
- Opening a file whose name has a mixed-case stem (e.g. AccountHistory.csv)
now works correctly on case-sensitive Linux filesystems. The controller
was lowercasing the stem before calling open_table, causing DataSource to
look for "accounthistory.csv" which does not exist. The stem is now
passed verbatim from the filesystem path; URL-sourced names (from
/view/:table and table: join specs) are still normalised to lowercase as
before because data/ files are lowercase by convention.
- CSV files whose first column header is not a valid SQL identifier (e.g.
"Account Number" from a bank export) now open without error.
DataSource::_detect_file_info previously passed the first column name
directly to Database::Abstraction as the id parameter; D::A validates id
against $SAFE_IDENTIFIER and croaked "unsafe id column name".
_detect_file_info now scans the header and picks the first column name
that IS a safe identifier.
- CSV files where the first safe-identifier column is always empty (e.g.
"Check" in a US bank statement) now show all rows. Database::Abstraction
uses empty_is_undef => 1, so an empty Check cell becomes undef and the
row-existence sentinel grep discards every row where Check is blank.
_detect_file_info now reads the first data row and skips safe-identifier
columns whose value there is empty, landing on a reliably populated column
(e.g. "Description") as the id.
- If no column header in a CSV satisfies the safe-identifier constraint,
DataSource now croaks error_no_safe_id with a clear message asking the
user to rename at least one column, rather than silently returning 0 rows.
[ Test Improvements ]
- Added targeted tests in t/function.t, t/path.t, and t/domain.t that
would have caught all three bugs above before they reached production:
unit-level _detect_file_info subtests for each id-selection scenario,
new CFG paths G-J in path.t, and end-to-end HTTP domain subtests that
assert all rows are visible for mixed-case stems, spaced column headers,
and empty sentinel columns.
0.003.0 Wed Aug 26 16:54:40 UTC 2026
[ Enhancements ]
- Platform and language are now detected per-request rather than read
statically from config. CGI::Info inspects the User-Agent header to
select 'mobile', 'tablet', or 'web'; CGI::Lingua selects the best
ISO 639-1 language from the Accept-Language header against the set of
languages that have a template directory under templates/<platform>/.
Config values (platform, language) remain as fallbacks for when a
module is unavailable, the header is absent, or no template directory
exists for the detected value.
- Add "Refresh" button (↻) to the toolbar next to Export. Pressing it
reloads the current view from disk, picking up any changes to the
source file(s) since the page was last loaded. All URL state (joins,
filters, dedup) is preserved because location.reload() reuses the
current URL.
- Add "Hide duplicates / Show duplicates" toggle button to the toolbar.
When active (filled blue), duplicate rows are removed from the displayed
table and from any CSV or SQLite export. The button text reflects the
current state: "Hide duplicates" when duplicates are visible, "Show
duplicates" when they are suppressed. State is carried in the ?d=1
query parameter so the URL remains shareable and bookmarkable.
- Added "Clear upload cache" button (POST /uploads/clear).
Mojo::File->list() requires {dir => 1} to
include subdirectories; without it the upload subdirs are silently
skipped and the action always reports 0 files freed.
[ Bug Fixes ]
- Files opened via the "recently saved" section are no longer also added
to the "recently opened" list. The saved-section card href now carries
a transient ?from=saved marker; the dashboard JS reads and removes it
via history.replaceState (keeping the address bar clean) and skips the
bi:recent write when the marker is present.
- Recently saved files on the home page now open with a single click;
double-click was required before, making the feature effectively
inaccessible on touch devices and to users who did not discover the
interaction. Missing files (greyed-out cards) still block navigation.
0.002.0 Wed Aug 26 08:21:12 EDT 2026
[ Enhancements ]
- Add "Recently saved" section to the home page: the last 5 files written
via POST /export appear as cards below "Recently opened". Each card
fetches /api/stat on page load -- missing files are immediately greyed
out and their cards block navigation. Mouseover shows saved-on date,
last-modified time, file size, and a "File not found on disk" warning
when the file has been moved or deleted.
- Add "Combine data" feature (GET /combine): stack rows from two or more
data files vertically into a unified view. All columns from all sources
appear as headers; cells are blank where a source file lacks that column.
Equivalent to a SQL UNION ALL across heterogeneous schemas.
- New "Combine data..." toolbar button (teal) sits next to "Merge data...";
both panels are mutually exclusive (opening one closes the other).
- Export pipeline (GET /export, POST /export) now handles c= params so
combined views can be downloaded or saved to disk.
- Internal helpers migrated from Sub::Private (:Private) to Sub::Protected
(:Protected) in Dashboard.pm and DataSource.pm for correct OO dispatch
in production (morbo/hypnotoad) without the CHECK-phase stash-deletion
side-effect that made :Private OO calls fail at runtime.
- "Merge data" button tooltip improved; join chip no longer shows key
annotation for combine summaries that have no join key.
0.001.0 Tue Aug 25 08:14:37 PM EDT 2026
- Initial CPAN release.
- Mojolicious web application presenting arbitrary flat data files
(CSV, PSV, XML, SQLite) as styled, sortable, reorderable HTML tables.
- Home page card grid; filesystem browser (/browse, /open).
- In-memory left-join engine (/join) with repeatable join steps.
- Server-side result filters (eq, ne, contains, starts, lt, le, gt, ge,
empty, notempty operators).
- CSV and SQLite export via GET /export (stream download) and
POST /export (filesystem write).
- Drag-and-drop file upload (POST /upload).
- JSON APIs: /api/columns, /api/dirs, /api/stat.
- URL import: fetch and display an HTML table from any public URL
(/import).
- localStorage persistence for column order and sort state.
Keyboard Shortcuts
Global
s
Focus search bar
?
Bring up this help dialog
GitHub
gp
Go to pull requests
gi
Go to GitHub issues (only if GitHub is preferred repository)