NAME

DiaColloDB::Relation::TDF - collocation db, profiling relation: (term x document) raw-frequency matrix

SYNOPSIS

##========================================================================
## PRELIMINARIES

use DiaColloDB::Relation::TDF;

##========================================================================
## Constructors etc.

$rel = CLASS_OR_OBJECT->new(%args);

##========================================================================
## TDF API: Utils

$vtype = $rel->vtype();
$itype = $rel->itype();
$packas = $rel->vpack();
$packas = $rel->ipack();

##========================================================================
## Persistent API: disk usage

@files = $rel->diskFiles();

##========================================================================
## Persistent API: header

@keys = $rel->headerKeys();
$hdr = $rel->headerData();

##========================================================================
## Relation API: open/close

$rel_or_undef = $rel->open($base);
$rel_or_undef = $rel->close();
$bool = $rel->opened();

##========================================================================
## Relation API: creation

$rel = $CLASS_OR_OBJECT->create($coldb,$tokdat_file,%opts);
$rel = CLASS_OR_OBJECT->union($coldb, \@dbargs, %opts);

##========================================================================
## Relation API: info

\%info = $rel->dbinfo($coldb);

##========================================================================
## Relation API: profiling

$mprf   = $rel->profile($coldb, %opts);
$mprf   = $rel->extend($coldb, %opts);
$mpdiff = $rel->compare($coldb, %opts);

##========================================================================
## Profile: Utils: PDL-based profiling

$mprf = $rel->vprofile($coldb, \%opts);

##========================================================================
## Profile: Utils: domain sizes

$NT = $rel->nTerms();
$ND = $rel->nDocs();
$NC = $rel->nFiles();
$NA = $rel->nAttrs();
$NM = $rel->nMeta();

##========================================================================
## Profile: Utils: attribute positioning

\%tpos = $rel->tpos();
\%mpos = $rel->mpos();

##========================================================================
## Profile: Utils: query parsing & evaluation

$idPdl    = $rel->idpdl($idPdl);
$tupleIds = $rel->tupleIds($attrType, $attrName, $valIdsPdl);
$ti       = $rel->termIds($tattrName, $valIdsPDL);
$ci       = $rel->catIds($mattrName, $valIdsPDL);

$bool          = $rel->hasMeta($attr);
$enum_or_undef = $rel->metaEnum($mattr);

$cats          = $rel->catSubset($terms);

\%groupby      = $rel->groupby($coldb, $groupby_request, %opts);

##========================================================================
## Relation API: default: query info

\%qinfo = $rel->qinfo($coldb, %opts);

DESCRIPTION

DiaColloDB::Relation::TDF is a DiaColloDB::Relation subclass for document-level co-occurrence frequencies using PDL to efficiently store and query a sparse underlying (term x document) frequency matrix via the PDL::CCS package.

Supports Boolean expressions over both term- and document-level conditions (the latter via DDC #has[ATTRIBUTE,VALUE] or #has[ATTRIBUTE,/REGEX/] syntax) as well as grouping via literal indexed term- and/or document-level attributes.

An earlier version of this module was implemented as DiaColloDB::Relation::Vsem ("vector-space distributional semantic index").

Globals & Constants

Variable: @ISA: DiaColloDB::Relation::TDF inherits from DiaColloDB::Relation.

Constructors etc.

new

$rel = CLASS_OR_OBJECT->new(%args);

%args, object structure:

##-- user options
base   => $basename,   ##-- relation basename
flags  => $flags,      ##-- i/o flags (default: 'r')
mgood  => $regex,      ##-- positive filter regex for metadata attributes
mbad   => $regex,      ##-- negative filter regex for metadata attributes
submax => $submax,     ##-- choke on requested tdm cross-subsets if dense subset size ($NT_sub * $ND_sub) > $submax; default=2**29 (512M)
mquery => \%mquery,    ##-- qinfo templates for meta-fields (default: textClass hack for genre): ($mattr=>$TEMPLATE, ...)
##
##-- logging options
logvprofile => $level, ##-- log-level for vprofile() (default=undef:none)
logio => $level,       ##-- log-level for low-level I/O operations (default=undef:none)
##
##-- modelling options (formerly via DocClassify)
minFreq    => $fmin,   ##-- minimum total term-frequency for model inclusion (default=undef:use $coldb->{tfmin})
minDocFreq => $dfmin,  ##-- minimim "doc-frequency" (#/docs per term) for model inclusion (default=4)
minDocSize => $dnmin,  ##-- minimum doc size (#/tokens per doc) for model inclusion (default=4; formerly $coldb->{vbnmin})
maxDocSize => $dnmax,  ##-- maximum doc size (#/tokens per doc) for model inclusion (default=inf; formerly $coldb->{vbnmax})
vtype      => $vtype,  ##-- PDL::Type for storing compiled values (default=float; auto-promoted if required)
itype      => $itype,  ##-- PDL::Type for storing compiled integers (default=long)
##
##-- guts: aux: info
N => $tdm0Total,       ##-- total number of (doc,term) frequencies counted
dbreak => $dbreak,     ##-- inherited from $coldb on create()
##
##-- guts: aux: term-tuples ($NA:number of term-attributes, $NT:number of term-tuples)
attrs  => \@attrs,       ##-- known term attributes
tvals  => $tvals,        ##-- pdl($NA,$NT) : [$apos,$ti] => $avali_at_term_ti
tsorti => $tsorti,       ##-- pdl($NT,$NA) : [,($apos)]  => $tvals->slice("($apos),")->qsorti
tpos   => \%a2pos,       ##-- term-attribute positions: $apos=$a2pos{$aname}
##
##-- guts: aux: metadata ($NM:number of metas-attributes, $NC:number of cats (source files))
meta => \@mattrs         ##-- known metadata attributes
meta_e_${ATTR} => $enum, ##-- metadata-attribute enum
mvals => $mvals,         ##-- pdl($NM,$NC) : [$mpos,$ci] => $mvali_at_ci
msorti => $msorti,       ##-- pdl($NC,$NM) : [,($mpos)]  => $mvals->slice("($mpos),")->qsorti
mpos  => \%m2pos,        ##-- meta-attribute positions: $mpos=$m2pos{$mattr}
##
##-- guts: model (formerly via DocClassify dcmap=>$dcmap)
tdm => $tdm,             ##-- term-doc matrix : PDL::CCS::Nd ($NT,$ND): [$ti,$di] -> f($ti,$di)
tym => $tym,             ##-- term-year matrix: PDL::CCS::Nd ($NT,$NY): [$ti,$yi] -> f($ti,$yi)
cf  => $cf_pdl,          ##-- cat-freq pdl:     dense:       ($NC)    : [$ci]     -> f($ci)
c2date => $c2date,       ##-- cat-dates   : dense ($NC)   : [$ci]   -> $date
c2d    => $c2d,          ##-- cat->doc map: dense (2,$NC) : [*,$ci] -> [$di_off,$di_len]
d2c    => $d2c,          ##-- doc->cat map: dense ($ND)   : [$di]   -> $ci
#...

TDF API: Utils

vtype

$vtype = $rel->vtype();

get PDL::Type value type for storing compiled values.

itype

$itype = $rel->itype();

get PDL::Type integer type for storing compiled indices.

vpack

$packas = $rel->vpack();

pack-template for $rel->vtype(), e.g. "f*"

ipack

$packas = $rel->ipack();

pack-template for $rel->itype(), e.g. "l*"

Persistent API: disk usage

diskFiles

@files = $rel->diskFiles();

returns disk storage files, used by du() and timestamp()

headerKeys

@keys = $rel->headerKeys();

keys to save as header; override includes qw(meta attrs vtype itype) and excludes logging and i/o keys.

headerData

$hdr = $rel->headerData();

returns reference to object header data; override stringifies {itype} and {vtype} keys.

Relation API: open/close

open

$rel_or_undef = $rel->open($base);
$rel_or_undef = $rel->open($base,$flags);
$rel_or_undef = $rel->open();

Opens underlying index files.

close

$rel_or_undef = $rel->close();

Closes underlying index files.

opened

$bool = $rel->opened();

Returns true iff index is opened. Really just checks for $rel->{tdm}.

Relation API: creation

create

$rel = $CLASS_OR_OBJECT->create($coldb,$tokdat_file,%opts);

Populates relation index for $coldb. Requires:

(temporary, tied) doc-arrays @$coldb{qw(docmeta docoff)}
temp file "$coldb->{dbdir}/vtokens.bin": pack($coldb->{pack_w}, @wattrs)

OR

wdmfile=>$wdmfile option

%opts: clobber %$rel, also:

docmeta =>\@docmeta, ##-- for union(): override $coldb->{docmeta}
                     ##   $docmeta[$ci] = {id=>$id, nsigs=>$nsigs, file=>$rawfile, date=>$date, label=>$label, meta=>\%meta}
wdmfile =>$wdmfile,  ##-- for union(): txt ~ "$ai0 $ai1 ... $aiN $doci $f"; default is generated from 'vtokens.bin'
ivalmax =>$imax,     ##-- for union(): maximum integer value (for auto-promotion)
reusedir=>$bool,     ##-- for union(): set to true if we're running in a "clean" directory
logas   =>$logas,    ##-- log label (default: 'create()')

union

$rel = CLASS_OR_OBJECT->union($coldb, \@dbargs, %opts);

merge multiple tdf indices into new object. \@dbargs is an ARRAY-ref of DiaColloDB sub-objects ($coldb,...) containing {tdf} relations to be merged.

%opts: clobber %$rel

Current implementation just creates temp-files utdm0.dat and udocmeta.tmp and then calls create().

Relation API: info

dbinfo

\%info = $rel->dbinfo($coldb);

embedded info-hash for $coldb->dbinfo()

Relation API: profiling

profile

$mprf = $rel->profile($coldb, %opts);

Get a relation profile for selected items as a DiaColloDB::Profile::Multi object. %opts are as for DiaColloDB::Relation::profile(). Really just a wrapper for the vprofile() method.

extend

Get independent f2 frequencies for $opts{slice2keys} as a DiaColloDB::Profile::Multi object.

compare

$mpdiff = $rel->compare($coldb, %opts);

Get a relation comparison profile for selected items as a DiaColloDB::Profile::MultiDiff object. %opts are as for DiaColloDB::Relation::compare(), which this method calls after parsing the groupby option via $rel->groupby($coldb, $opts{groupby}, relax=>0).

Profile: Utils: PDL-based profiling

vprofile

\@pprfs = $rel->vprofile($coldb, \%opts);

Guts for the profile() method. User options in %opts are as for DiaColloDB::Relation::profile(). Additional keys are populated and used in the course of the computation (so don't set them):

vq      => $vq,        ##-- parsed query, DiaColloDB::Relation::TDF::Query object
groubpy => \%groupby,  ##-- as returned by $rel->groupby($coldb, \%opts)
dlo     => $dlo,       ##-- as returned by $coldb->parseDateRequest(@opts{qw(date slice fill)},1);
dhi     => $dhi,       ##-- as returned by $coldb->parseDateRequest(@opts{qw(date slice fill)},1);
dslo    => $dslo,      ##-- as returned by $coldb->parseDateRequest(@opts{qw(date slice fill)},1);
dshi    => $dshi,      ##-- as returned by $coldb->parseDateRequest(@opts{qw(date slice fill)},1);

Profile: Utils: domain sizes

nTerms

$NT = $rel->nTerms();

returns number of indexed terms.

nDocs

$ND = $rel->nDocs();

returns number of indexed documents (breaks).

nFiles

$NC = $rel->nFiles();

returns number of indexed categories (original source files).

nAttrs

$NA = $rel->nAttrs();

returns number of indexed term-attributes.

nMeta

$NM = $rel->nMeta();

returns number of indexed meta-attributes.

Profile: Utils: attribute positioning

tpos

\%tpos = $rel->tpos();
$tpos  = $rel->tpos($tattr);

In the first form, get or build the term-attribute position lookup hash. In the second form, get the index position along dimension $NA of the term-attribute named $tattr, or undef if $tattr is not a known term attribute.

mpos

\%mpos = $rel->mpos();
 $mpos  = $rel->mpos($mattr);

In the first form, get or build the meta-attribute position lookup hash. In the second form, get the index position along dimension $NM of the meta-attribute named $mattr, or undef if $mattr is not a known metadata attribute.

Profile: Utils: query parsing & evaluation

idpdl

$idPdl = $rel->idpdl($idPdl);
$idPdl = $rel->idpdl(\@ids);
$idPdl = $rel->idpdl($id);

Ensure PDL-ness of a set of integer IDs.

tupleIds

$tupleIds = $rel->tupleIds($attrType, $attrName, $valIds);

Returns a PDL representing the set of index items of type $attrType whose value for the $attrName attribute is contained in the ID-set $valIds, which may be specified in any of the forms accepted by the idpdl() method.

$attrType is either 't' for a term-attribute (in which case the returned $tupleIds are term indices), or 'm' for a metadata attribute (in which case the returned $tupleIds are "category" indices). The returned $tupleIds are always sorted in ascending order.

Could use some optimization.

termIds

$ti = $rel->termIds($tattrName, $valIds);

wraps $rel->tupleIds('t',$tattrName,$valIds).

catIds

$ci = $rel->catIds($mattrName, $valIds);

wraps $rel->tupleIds('m',$mattrName,$valIds).

hasMeta

$bool = $rel->hasMeta($mattr);

returns true iff $rel supports metadata attribute $mattr.

metaEnum

$enum_or_undef = $rel->metaEnum($mattr);

returns metadata attribute enum for $attr, or undef if $mattr is not supported.

catSubset

$cats = $rel->catSubset($termIds);
$cats = $rel->catSubset($termIds,$catIds)

Get a (sorted) cat-subset for the (sorted) term-set $termIds: the set of all "categories" (original source files) which contain at least one instance of any of the terms in $termIds, optionally restricted to the (sorted and unique) set $catIds. The returned category-IDs are sorted and unique.

groupby

\%groupby = $rel->groupby($coldb, $groupby_request, %opts);
\%groupby = $rel->groupby($coldb, \%groupby,        %opts);

Modified version of DiaColloDB::groupby() suitable for pdl-ized TDF relation. $grouby_request is as for DiaColloDB::parseRequest(). Returns a HASH-ref:

##-- COMPAT: equivalent to DiaColloDB::groupby() return values
req => $request,    ##-- save request
areqs => \@areqs,   ##-- parsed attribute requests ([$attr,$ahaving, \%ainfo],...)
                    ##   + new: %ainfo = ( aname=>$enum_name, atype=>$t_or_m, apos=>$apos )
attrs => \@attrs,   ##-- like $coldb->attrs($groupby_request), modulo "having" parts
titles => \@titles, ##-- like map {$coldb->attrTitle($_)} @attrs
##
##-- NEW: for DiaColloDB::Relation::TDF
how      => $ghow,     ##-- one of  't':groupby terms-only, 'c':groupby cats-only, 'tc':groupby terms+cats
gatype   => $gatype,   ##-- pdl ($NG)         : attribute types $ai : 0 if $areqs->[$ai] is a term attribute, 1 if meta-attribute
gapos    => $gapos,    ##-- pdl ($NG)         : term- or meta-attribute position indices $ai : $rel->mpos($attrs[$ai]) or $rel->tpos($attrs[$ai])
ghavingt => $ghavingt, ##-- pdl ($NHavingTOk) : term indices $ti s.t. $ti matches groupby "having" requests, or undef
ghavingc => $ghavingc, ##-- pdl ($NHavingCOk) : cat  indices $ci s.t. $ci matches groupby "having" requests, or undef
g2s      => \&g2s,     ##-- stringification object suitable for DiaColloDB::Profile::stringify() [CODE,enum, or undef]
gpack    => $packas,   ##-- pack template for groupby-keys

%opts:

warn  => $level,    ##-- log-level for unknown attributes (default: 'warn')
relax => $bool,     ##-- allow unsupported attributes (default=0)

Relation API: default: query info

qinfo

\%qinfo = $rel->qinfo($coldb, %opts);

get query-info hash for profile administrivia (ddc hit links). %opts: as for profile() method. returned hash \%qinfo should have keys:

fcoef     => $fcoef,     ##-- frequency coefficient (constant 1 here)
qtemplate => $qtemplate, ##-- query template with __W1.I1__ rsp __W2.I2__ replacing groupby fields

AUTHOR

Bryan Jurish <moocow@cpan.org>

COPYRIGHT AND LICENSE

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.14.2 or, at your option, any later version of Perl 5 you may have available.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)