/ 
DBIO-ClickHouse-0.900004
/ DBIO::ClickHouse::Native

NAME

DBIO::ClickHouse::Native - Turn DBIO schemas and rows into ClickHouse::Encoder input

VERSION

version 0.900004

SYNOPSIS

use DBIO::ClickHouse::Native;
use ClickHouse::Encoder;   # yours to load, never ours -- see below

# 1. The column list, straight out of the DBIO source
my $cols = DBIO::ClickHouse::Native->columns_for($schema->source('Hit'));
# [ ['id','UUID'], ['url','String'], ['hits','UInt32'],
#   ['snapshot','Nullable(Int32)'], ['ts','Nullable(DateTime64(3))'] ]

# 2. The rows, in that column order, deflated
my $rows = DBIO::ClickHouse::Native->rows_from(
  $schema->resultset('Hit')->search({ hits => { '>' => 100 } }), $cols
);

# 3. Hand both to ClickHouse::Encoder. Note the port: 8123 is the HTTP
#    interface, a different channel from the driver's MySQL wire on 9004.
my $resp = ClickHouse::Encoder->insert_http(
  host    => '127.0.0.1',
  port    => 8123,
  table   => 'dbio_test_hit',
  columns => $cols,
  rows    => $rows
);
die "insert failed (status $resp->{status}): $resp->{content}"
  unless $resp->{success};

A large export should not be materialized. "row_iterator" returns exactly the coderef ClickHouse::Encoder->stream pulls from:

my $enc  = ClickHouse::Encoder->new(columns => $cols);
my $iter = DBIO::ClickHouse::Native->row_iterator($rs, $cols);

open my $fh, '|-', 'clickhouse-client',
  '--query', 'insert into dbio_test_hit format native' or die $!;
$enc->stream($iter, sub { print {$fh} $_[0] }, batch_size => 10_000);
close $fh or die 'clickhouse-client failed';

DESCRIPTION

DBIO::ClickHouse::Native translates between two data representations: the DBIO side (a DBIO::ResultSource and its rows) and the input ClickHouse::Encoder takes (an arrayref of [$name, $type] pairs, plus rows as arrayrefs in that column order). It produces plain Perl data structures and nothing else.

It is not a transport, and it never loads ClickHouse::Encoder. There is no use, no require, and no runtime probe for it anywhere in this module (t/50-native.t asserts that via %INC; only the optional t/51-native-encoder.t loads the encoder); you load the encoder yourself and drive encode / insert_http / bulk_inserter from your own code. That is what keeps the dependency a suggests rather than a requires -- see docs/adr/0007-native-format-export-bridge.md -- and it is why this layer does not collide with ADR 0001: the MySQL wire protocol on port 9004 remains this driver's only transport, and the Native format goes out over a channel DBIO has no part in.

It is one-way: DBIO to encoder. There is deliberately no counterpart that turns ClickHouse::Encoder->decode_rows output back into a $rs->populate. Reading already comes back over the MySQL wire, so the Native format would buy no throughput on that side, and supporting it would duplicate the value-conversion rules below for a much narrower case.

The column list is our truth, not the table's

"columns_for" answers "what type would this distribution deploy for this column" -- it is "column_type" in DBIO::ClickHouse::DDL, the same renderer behind CREATE TABLE and the diff. If the target table was created by this distribution's deploy, the two agree by construction.

If it was not -- created by hand, by another tool, or by an older version of your schema -- it can differ, and the Native format is typed: a UInt32 column list against an Int64 table column is a wrong-width block, which is a server error at best. When the live table is the authority, ask it instead:

my $enc = ClickHouse::Encoder->for_table('dbio_test_hit', host => 'db');

for_table runs describe table and builds the encoder from the answer. You can still use "rows_from" with $enc->columns as the column list.

Composite types are a one-way street

Array(T), Map(K, V), Tuple(...), JSON, Variant(...) and the geo aliases need a Perl reference per value. This distribution happily renders those types at deploy time, but reading such a column back over the MySQL wire protocol yields a string -- ClickHouse's text rendering of the value, ['perl','db'] -- not an arrayref. There is no round trip.

So a composite column can only be exported from rows you built yourself. This module does not paper over that with heuristic string splitting; when "rows_from" meets a composite column holding a plain scalar it throws, naming the column and the reason. The encoder would refuse the value too (Expected arrayref for Array type) -- what is added here is the diagnosis, because the interesting question is not that the value is a string but why. See "LIMITATIONS".

Values are deflated, not inflated

Rows go through "get_column" in DBIO::Row, which is DBIO's deflation path: a value set through an InflateColumn accessor is deflated there and handed back in its stored form. A DateTime object from DBIO::InflateColumn::DateTime therefore arrives as the YYYY-MM-DD HH:MM:SS string the storage's datetime parser produced -- which is what a DateTime, DateTime64(P), Date or Date32 column in ClickHouse::Encoder parses. Rows read from the database are already in that form, because that is what the MySQL wire hands over.

What this module does not do is convert values itself. If a blessed object reaches it, deflation did not happen, and it throws rather than passing it on. That is not a formality. ClickHouse::Encoder does not dispatch through ->epoch; it takes the value's string or numeric conversion, so an object encodes to something -- usually to something wrong, and quietly. Measured against ClickHouse::Encoder 0.02 with a DateTime->new(..., time_zone => 'Europe/Berlin'):

  • into DateTime: the object stringifies to ISO 8601 without a zone marker, so it is read as UTC and the encoded epoch is off by the zone's offset. No error, wrong timestamp.

  • into Int64 or Float64: 2026 -- the leading digits of 2026-08-11T12:00:00. No error.

  • into Decimal(10, 2): 202600. No error.

The fix is on the DBIO side: register the inflator/deflator for the column.

Passing the stored value through untouched is not merely tidy, it is load bearing. A Decimal* column arrives from the DBD as a string, and ClickHouse::Encoder parses a decimal string digit by digit and scales it exactly, where a decimal number goes through a double and is lossy past 2^53. Numifying a value on its way here -- in this module or in your own code -- silently drops digits.

METHODS

columns_for

my $cols = DBIO::ClickHouse::Native->columns_for($source);
my $cols = DBIO::ClickHouse::Native->columns_for($source,
             columns => [qw( id ts kind )]);

Returns an arrayref of [$name, $type] pairs for ClickHouse::Encoder->new(columns => $cols). $type comes from "column_type" in DBIO::ClickHouse::DDL.

The first argument may be a DBIO::ResultSource, a DBIO::ResultSet, a result object, or a result class name (loaded on demand). Anything else throws rather than being guessed at.

Without columns every column of the source is emitted, in the source's own order. With columns only the named ones are, in the order given -- the list you pass is the block layout, so it is also the order "rows_from" produces. An unknown column name throws.

rows_from

my $rows = DBIO::ClickHouse::Native->rows_from($rs, $cols);
my $rows = DBIO::ClickHouse::Native->rows_from(\@results, $cols);

Returns an arrayref of arrayrefs -- one per row, values in $cols order -- ready for $enc->encode($rows), insert_http(rows => $rows) or $bi->push_many($rows).

$cols is required and is normally what "columns_for" returned; passing it explicitly is what makes the encoder and the rows provably agree on the block layout. The first argument is a DBIO::ResultSet (consumed with next, exactly like a while loop would) or an arrayref of rows; a row is either a DBIO result object or a plain hashref, so a resultset switched to DBIO::ResultClass::HashRefInflator works and is the faster path for a large export.

This slurps the whole result into memory. For a big export use "row_iterator" instead.

Throws on anything it cannot export truthfully: a column the row never fetched, a NULL in a column whose type is not Nullable, a blessed value (see "Values are deflated, not inflated"), or a composite column holding a plain scalar (see "Composite types are a one-way street").

row_iterator

my $iter = DBIO::ClickHouse::Native->row_iterator($rs, $cols);
while (defined(my $row = $iter->())) { ... }

Same conversion as "rows_from", one row at a time: returns a coderef that yields the next row arrayref and undef when the source is exhausted. That is precisely the iterator contract $enc->stream(\&iter, \&writer) expects, so an export of any size can go straight into the encoder's streaming path without ever holding more than one batch.

LIMITATIONS

  • No decode side. DBIO to encoder only; see "DESCRIPTION".

  • Composite columns cannot be read back out of ClickHouse. Array, Map, Tuple, JSON, Variant and the geo types come back over the MySQL wire as their text rendering. Export them from data you built in Perl; "rows_from" throws rather than guessing at a string.

  • The column list is this distribution's deploy truth. Use ClickHouse::Encoder->for_table when the live table is the authority.

  • Nothing here validates against the server. The types are rendered from schema metadata; whether the target table matches is found out at insert time, by the encoder or by ClickHouse.

SEE ALSO

AUTHOR

DBIO Authors

COPYRIGHT AND LICENSE

Copyright (C) 2026 DBIO Authors

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.