NAME

Text::Fab - A powerful, general-purpose document expansion framework (currently Caching is unimplemented {???})

SYNOPSIS

Three input files, written for the default (dbg0) callbacks:

## -- File: config.fab --
## Declare which configuration keys are lists
#append Fab/list_keys chapters 1
#append Fab/list_keys joiners/comma 1
## Set some configuration variables
#set site_title=My Awesome Book
#append chapters chapter1 chapter2 chapter3
#append joiners/comma ,
## Define the inheritance graph of the namespaces
#set_parents ChapterLayout BaseLayout
## The final book inherits from _main's structure and from ChapterLayout
#set_parents Book _main ChapterLayout

## -- File: components.fab --
## Reusable components, defined in namespaces
#target_section title in ChapterLayout
<h1>
#include configkey joiners/comma site_title
</h1>
#end_section
#target_section table_of_contents in ChapterLayout
<p>Chapters:
#include configkey joiners/comma chapters
</p>
#end_section
#target_section footer in BaseLayout
<footer>Copyright 2025</footer>
#end_section

## -- File: main.fab --
## The main assembly script.  The default output target is _main:body
#emb title
#emb table_of_contents
#emb footer

Processing them, then assembling the book:

use Text::Fab;
my $fab = Text::Fab->new;
$fab->include__text(filename => $_) for qw(config.fab components.fab main.fab);
print $fab->assemble('Book:body'), "\n";

prints

<h1>
My Awesome Book</h1>
<p>Chapters:
chapter1,chapter2,chapter3</p>
<footer>Copyright 2025</footer>

Book:body is not defined, so it is found in _main, the first parent of Book. The promises inside it are resolved late, starting from Book, the namespace the body was requested in: title and table_of_contents are found in ChapterLayout, and footer in BaseLayout (the parent of ChapterLayout). #include configkey runs while the file is read, so the configuration it uses must be set before (here: in config.fab).

Not yet implemented {???}: the original version of this example also used the constructs below. The default callbacks do not support them (yet); they are kept here as a reminder of the intended syntax.

#append Fab/list_keys chapters css_files joiners/lines  # several names at once, trailing comment
#append joiners/lines "\n"                              # a quoted value with an escape
<h1>#emb _main:title</h1>                               # a directive inside a line: needs a custom interleaver
#target_section all_chapters in ChapterLayout
    #= FOREACH chapter IN C('chapters') ...             # needs a Chunk Preprocessor that understands "#="
        #emb @chapter:body
    #= END #
#end_section

DESCRIPTION

Text::Fab is not a templating engine; it is a framework for building your own. At its core, it is more or less a minimal "dumb" state machine that exposes a stable API of primitive operations. Most of the "smart" logic for parsing syntax, processing text, and creating complex control structures is delegated to user-configurable components. (However, a certain minimal level of smartness is a — switchable off — default to handle the most common cases.) As a result, it knows nothing about the processed “language”'s syntax or semantics.

The main design goal is that all state of Fab is explicitly introspectable: there is no hidden magic. The entire state of the system is contained in the Configuration hash and the collection of already constructed Output Sections. What you see is what you get.

The Fab’s purpose is to factor out the common needs of all the configurable-on-the-fly document processing engines; more precisely, it focuses on the needs orthogonal to the particular syntax of a particular problem domain. (This architecture aims to overcome the experiences with designing “simple” configurable systems — which turn out to be not scalable, so all the approaches to make them operable crumble under their own weight.)

In short: this attempts to abstract out all the complexity of scalable configurability. The resulting “tools” support architecture with a top→down approach to recognizing the “reconfiguration directives”.

————————————————————————————————————

By default, Text::Fab reads input and filters it through a specified preprocessing engine (which defaults to pass-through-unchanged) to the standard output. Its power comes from executing the interleaved-in-the-input directives that are tools for controlling the processing engine, reshuffling content, and templating.

The process is straightforward: Text::Fab asks a De-Interleaver callback "Where is the next control directive?". It then (optionally, see below) passes the text chunk before that location to a Chunk Preprocessor (the "stomach"). Finally, it allows the Parser to execute the directive (typically, the Parser would convert the details of the directive into calls to zero or more of Text::Fab's primitive API methods to alter its internal state). This cycle repeats until the input is exhausted.

Furthermore, as an extra configurable step, the De-Interleaver may need to report that a particular directive “is a comment-to-ignore”. In this case, Fab would postpone processing the text chunk before that location, instead merging it with the following chunk(s) before passing to the Pre-Processor. So the process’ cycle is: the De-Interleaver finds the next directive. The Comment Recognizer may then optionally identify a comment, allowing for the fusion of the preceding and succeeding text chunks. Depending on this: now — or later — this final chunk is then passed to the Chunk Preprocessor. Finally, the Parser executes the directive found above, which may alter the system's state for the next cycle.

Finally, this “structured” (see below) pre-processed data created this way is “merged” into the final output text stream following flexible “hierarchical” rules.

Preview of architecture

To explain what this engine is supposed to simplify, consider a (somewhat contrived) example of document preparation. You have a prepared source of a document; it may consists of several files which should be joined by certain rules; each file is a mix of (recognizable) chunks of markup, LaTeX and HTML; etc. You want to produce a best-effort PDF file, several LaTeX files prepared for publication in various outlets, a readable plain-text version, and a web page.

To add insult to injury, many parts of the document should be displayed differently in different parts of the output. (For example, a section header may be quadriplicated: as a TOC entry, a running page header, as a PDF’s TOC entry, and its rendition in the main text.) And the details of prepocessing some embedded promise (say, MathChunk with the “content” x²) may need to be “massaged” into (the same) string x² for the PDF header, or to $x^2$ when handled inside TOC (or running headers), or to $$x^2$$ in the header rendered “as a part of main text” (well, I warned you that the example is contrived!).

On top of this, the rendition may depend on a mix of recipes inherited from different sources: the default solution “from the vendor”, the enhancements designed by your organization, enhancements found on the web, the requirements of the journal, the legal requirements of the country, your personal enhancements to these etc.

How to handle processing of all this mess through one pass over the initial document? The architecture of Text::Fab encodes the intermediate steps of processing into 6 components, One of them — The Configuration — is very special since all other components (except the content of output sections) are in fact encoded inside it. The aim of these components is to provide carpets under which one can conveniently sweep all the parts of complexity-dust which “require coding” — leaving only “the elementary core dependencies” exposed to both the user and the developer. The intent is that after this sweeping, all (or: better be all?) the necessary configuration/fine-tuning might be done not by writing/editing new code, but by issuing processing directives.

Configuration: A multi-level hash with all the control data needed to process the flow, as well as the temporary variables.
Undo engine: The (structured) list of undo actions to execute at the end of certain processing steps (possibly including undoing changes to the configuration).
Input chain: The inclusion-chain of input files currently in processing (but not yet finished).
Output sections: They are named, and encode chunks of semi-digested input. Each consists of plain text strings and of “promises” to be resolved later, all in one (1-dimensional) list. Each promise contains a name of recipe and the name of section to process (and possibly hints on the “name resolution” of recipes).
Namespaces: They form a DAG encoding “which namespace overrides what” (with the root overriding them all, etc.). Each one resolves certain recipe names to the code to implement these recipes. (Note that if A overrides B, this may be interpreted as “B contains the default actions for A”. — Following the metaphor of inheritance, we call B “a parent” for A.)
A chain of “lenses”: Each lens encodes “the aim of the currently active pipeline” of processing the promises in the sections. Each request to set a new aim is translated (see "Massaging of lens names") to just a particular vertex in the DAG of namespaces, a “virtual root” — so the name resolution is restricted to it and “its defaults”: whatever namespaces it overrides. The resulting vertex is appended the chain of lenses.

Example of usage

With our still contrived example, suppose that the initial pass over the document converted to an output section §base with 4 promises (and nothing else):

#emb secname §sec1name
#emb sectext §sec1text
#emb secname §sec2name
#emb sectext §sec2text

(with §sec1name etc. being the names of other sections), and 4 “referenced above” sections — as well as some other sections which may be referenced from those 4 — etc. (These 4 sections are minimally preprocessed — just enough to recognize what region in the input files goes where (but one can as well also run the preprocessing steps common for all output formats). When processing to LaTeX: one of the first calls would establish the outermost lens to be LaTeX. When one needs to produce TOC, one processes §base as

#emb[aim=as_TOC_lens] as_TOC §base

The installed “recipes for lenses” would (presumably) change the requested root as_TOC_lens to LaTeX/as_TOC_lens. The resulting stack of lenses is now LaTeX LaTeX/as_TOC_lens. And the active lens LaTeX/as_TOC_lens is (presumably) set up to inherit from as_TOC_lens — which sets the name resolution to process sectext promises to completely ignore the corresponding output sections (which are now read for input!), while processing sections of secname promises into their “maximally compact output format”. And on top of this, LaTeX/as_TOC_lens would also arrange to wrap the result into the necessary LaTeX macros.

Should not we list how exactly it is done ???  A promise C<MathChunk> inside §sec2name would be (presumably) resolved to a
C<LaTeX/as_TOC_lens/MathChunk> recipe.  This recipe would pass the content (HOW???  How can it delegate to
C<as_TOC_lens/MathChunk>?  By another C<aim=>?) into a promise C<compact/MathChunk>, and wrap the result into a suitable
LaTeX macro.

One processes to PDF_TOC likewise — however, here one needs to transform from the content of §sec2name not to LaTeX macros, but to Unicode plain text. — This may be arranged as above, through a certain installed sub-preprocessor used as a pipe in “an output section” → “an output section”.

When one processes “the main body” of the output, one calls

#emb[aim=as_BODY_lens] as_BODY §base

which would lead (likewise) to the output sections of both secnames and sectexts to be fully processed (with suitable macros wrapping section names — which are also processed into a compact form to be put into the running headers).

Note that how the fact “that the output is to LaTeX” is set up by one top-level call (inside a certain 1-liner output section — WHICH???)! If one wants conversion to PDF or HTML instead and the necessary translators of ”elementary subblocks” are already present and configured, all that one needs to do is to replace this 1-liner by another one. Due to a change to the toplevel lens, the names of the following lenses on the stack would change accordingly (although the specified “aims” when installing these lenses did not change!) — which would change the namespace resolution “as needed”, so the same promises have a chance to depend completely differently. In particular, they may insert “the same promises as for LaTeX” (but which may now depend differently), or they may replace/enhance them by absolutely different promises specialized for HTML processing.

When one need to reformat the LaTeX output for a particular journal, one may override how the names of promises are converted to their “implementation” by including the needed implementations/enhancements into a new namespace(s), inserting them accordingly into the DAG of namespaces.

Massaging of lens names

The translation aim → lens above defaults to id (then the names of lenses conincide with “the request”). More generally, it depends on the current chain of lenses — controlled on the registered list of “which lenses affect which”. If a lens Foo affects Bar, then the result is the lens Foo/Bar; additionally, in the longer chain A, B, Foo, C, D, Bar, if A affects Foo/Bar, then the result is¹⁾ A/(Foo/Bar).

¹⁾ Caveat usor: in the chain C<A, Foo, Foo/Baz> a request for C<Bar> may result in²⁾ C<(Foo/Baz)/Bar>.
   (May later propagate to C<((Foo/Baz)/Bar)/Gee>.)  Since it is only the user/developer who could declare such “affections”,
   it is their responsibility to insert
²⁾ May we ever need to clean C<Foo/(Foo/Baz)/Bar>???

TODO: DAG affecting affections — need inheritance on Foo above???

TODO: If a new aim PDF3.0 appears, and one declared that it affects PDFbookmarks — which has been affecting TeXmath: if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is declared, then it is fine: the request to TeXmath inside such a chain would still create PDFbookmarks/TeXmath — which (hopefully) would be processing things correctly. (And if not, then it is the responsibility of whoever implemented this “new aim” to recode how TeXmath behaves in presence of PDF3.0/PDFbookmarks, and declare the corresponding “affects” declaration).

On the othe hand, if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is not declared (this may happen for example when the recipes in this namespace are ), then to enable the transparent handling (as above) one should use the other mechanism for “affecting to PDFbookmarks/TeXmath” (now with TeXmath “being a joker”): declare ???

THE PROCESSING MODEL

First of all, the inclusion mechanism handles a stack of currently opened-for-processing files; these files are “combined as usual” into one input stream (but each has its own control stack, see "or loops)"" in "Support of Control-Flow-Like constructs (conditionals and). The active file chain is reflected in the configuration key opened_includes. In addition to this, the Fab uses three more data collections.

The configuration directives may affect how the preprocessor handles the chunks of its input, whether “the comments interleaved into the stream” split these chunks etc., and may modify how the “directives interleaved into the input stream” are recognized (and even what these directives mean). The grouping directives control which changes to configuration are undone and when. The output directives may insert into the output section "an order to inline another section" (a promise, which is not executed until much later). Another kind of output directive allows switching the target section for the output of the preprocessor.

(It makes sense to also focus on specific parts of the state: the Control Stack which manages conditional handling and looping (kept beside the Configuration Hash, see "or loops)"" in "Support of Control-Flow-Like constructs (conditionals and), Grouping Stack which handles undoing changes to configuration, and Caching Engine which helps to avoid the penalties of constantly recalculating data dependent only on the configuration stack [Planned].)

After all the sections have been constructed, they may be “joined arbitrarily” into the final document. This assembly process is controlled by the rules how to choose a named section if “multiple flavors” have been defined. This allows a flexible hierarchical system of “overridable templates”. Such templates together with the rules of “namespace resolution” build the logic of processing that ultimately generates the final document(s).

The Configuration Hash

The heart of Text::Fab is a single Configuration hash. It is a live data structure that directives can modify during processing. It controls everything, from user-defined variables to the very components (parsers, etc.) that define the language's behavior. By default, all internal configuration used by the framework itself is stored under the Fab/ top-level key. (Later we are going to discuss how to modify this default, and how this prefix changes during the input processing. However, in examples below we assume that this hasn't been changed.)

Configuration values can be scalars, lists, or hashes. Special hashes, Fab/list_keys and Fab/hash_keys, are used to declare which keys should be treated as lists or hashes by the primitive operations. new() registers Fab/hash_keys, Fab/list_keys and Fab/lens_affects as hash keys.

Fab's own settings are read through the stack of prefixes Fab/call_stack_prefixes (see "Configuration Prefix and Call Stack API"): for a setting such as list_keys, Text::Fab looks at PREFIX/list_keys for each prefix on the stack, the most recent prefix first. This applies to list_keys, hash_keys and lens_affects (here a name is looked up in each prefix's hash in turn, so a recursive filter can add or override single entries), and to hash_key_sort_order, callbacks, callbacks_scope, callbacks_bundle/*, inheritance_graph, group_flavors, group_nesting, blinders/*, max_recursion_depth, max_include_depth, control__loop_limits, output_handle and output_postpone. NS__set_parents writes under the most recent prefix. Only state kept by Text::Fab itself has fixed names: the stack of prefixes, Fab/call_stack_prefixes and Fab/call_stack_filter_name, and the stack of output targets, Fab/output_target_stack.

The Output Sections

A section is a named buffer that holds already preprocessed content. A section's name is a pair of strings: a namespace and a basename (e.g., Chapter1:body). The default namespace is _main.

It makes sense to imagine that a section consists of two interleaved lists, each of its specific type of content:

Plain text chunks, which have been processed by the Chunk Preprocessor.
embed placeholders, which are promises to inline another section during the final Assembly Phase. This other section is described by its basename as well as some extra data (these data are used in the calculation of its namespace, which is going to be performed later).

Target Section Stacking and Scoping

#target_section <basename [in <namespace>]>: Switches the active output target to the specified section, pushing it onto the stack of targets (see "Output" below). If namespace is omitted, defaults to the current target's namespace (_main if there is none). Autovivifies the section buffer if it does not already exist. The basename and the namespace are arbitrary non-space tokens, so lens namespaces such as LaTeX/as_TOC or (LaTeX/as_TOC)/Math can be used; anything else on the line halts with E_SYNTAX_ERROR.
#end_section: Pops the current target from the stack, restoring the output stream to the enclosing target. On an empty stack, and on the base entry of postponed output, it does nothing.
#create_section <basename [in <namespace>]>: Initializes or clears the specified section buffer without redirecting current preprocessor output.

Output

The output targets are kept as a stack in the Configuration hash, Fab/output_target_stack: a list of [$namespace, $basename], whose last entry is the current target. Preprocessed text and promises (#emb) are appended to the current target section. Since the stack is part of the configuration, #end_group restores it like any other key: a group with a full snapshot restores it, and a flavor with a list of keys restores it if the list names Fab/output_target_stack.

When the stack is empty, there is no target section: text goes directly to the output, the filehandle Fab/output_handle (by default \*STDOUT; it must be a glob reference, such as \*STDOUT, a lexical handle from open, or an IO::Handle object, since FreezeThaw cannot snapshot a bare *FH{IO}). A promise, however, can only be resolved during assembly, after all the input has been read; so it cannot go to direct output. The setting Fab/output_postpone decides what happens when such a promise is going to be written:

at_first_promise (the default): Text goes directly to the output until the first promise meets the empty stack; then _main:body is pushed as the base entry of postponed output, and the promise and all later output go there. So, with the default callbacks, include__text() of a file without directives works like cat.
at_start: Output is postponed from the start: when output is produced with an empty stack, the target _main:body is pushed as the base entry of postponed output, and everything goes there. (This was the only behaviour in 0.08 and earlier.)
error: Text goes directly to the output; a promise meeting the empty stack halts with E_PROMISE_IN_DIRECT_OUTPUT. (Promises inside a target section are fine.)

#end_section (out__end_section()) never pops the base entry of postponed output. The postponed output is printed to Fab/output_handle by out__flush(), which a driver calls when all the input has been read. (Both settings may be changed in the input, e.g. #set Fab/output_postpone=error, as long as this happens before the first output.)

Grouping: Scoped/Undoable Configuration Changes

(Below, we assume a particular #-based format for directives. The actual operation is agnostic of this format.)

#start_group [flavor_name]

Begins a scope. A certain set of keys in the Configuration hash are snapshotted. The flavor_name is used to look up the list of Configuration keys to memorize/restore under Fab/group_flavors/$flavor_name/keys (an array reference). If this key list is not defined for the flavor, the default behavior is to snapshot the entire Configuration hash. With a key list, only these keys are restored by #end_group; all other changes survive the group. The key list can be set up with directives, too (here only x is restored):

#append Fab/list_keys Fab/group_flavors/g/keys 1
#append Fab/group_flavors/g/keys x
#start_group g
#set x=2
#set y=2
#end_group g
Validates nesting constraints against the enclosing group flavor via
C<Fab/group_nesting/$flavor/allowed_parents>, halting with C<E_INVALID_GROUP_NESTING> if violated.
The check is also made at top level: there, as inside an unnamed group, the enclosing flavor is
the empty string, so C<''> in the list allows both.
The active group flavor chain is mirrored in C<opened_groups>.
This key and C<opened_includes> always describe the groups and files open at the moment:
the configuration restored at C<#end_group> does not change them, even when the group
started in another file.

The option [callbacks=NAME] switches the set of callbacks with which Fab reads its input to the bundle Fab/callbacks_bundle/NAME (it sets Fab/callbacks). From where on the new set is used depends on Fab/callbacks_scope: nested (the default) uses it only for input that Fab starts reading later, e.g. files included inside the group (and, [Planned] {???}, promises expanded with their own callbacks); the rest of the current file keeps its set. next_line uses it from the line after the directive: Fab looks up the callbacks again after every directive line, so this also applies to any other change of Fab/callbacks. Any other value halts with E_TYPE_MISMATCH.

#end_group [flavor_name]

If a flavor_name is provided, it must match the flavor of the currently open group; otherwise, processing halts with E_MISMATCHED_END_GROUP. All changes made to the observed configuration keys are undone. If the optional name is not given, closes the most recently opened scope. Calling #end_group with no open scope halts with E_DANGLING_END_GROUP. Executes all postponed undo actions and restores the configuration snapshot.

Note: The nesting behavior of groups (e.g., whether flavor A must be nested within flavor B) is itself controlled by the Configuration hash.

DEFAULT (`dbg0`) DIRECTIVES REFERENCE

The default parser (_default_directive_parser) recognizes the following directives interleaved into the input stream.

Syntax. With the default callbacks, a directive is a line that starts with #: the interleaver recognizes # only as the first character of a line, so a # elsewhere (as in <h1>#emb title</h1>) is ordinary text. The directive line has the form #name or #name[options], followed by arguments separated by whitespace. There is no quoting, and a # among the arguments does not start a comment: it is passed on as an argument. A line that starts with ## is a comment. A line that starts with # but does not have this form (for example # text or #= ...) is reported to the error handler as malformed_directive. (Other syntaxes need other callbacks; their signatures are listed under PLACEHOLDERS below {???}.)

#set <key=<value>>

Sets a scalar configuration variable.

#append <key <values...>>

Appends values to a list key, appends string values to a scalar key, or merges an even-sized list of key-value pairs into a hash key.

#prepend <key <values...>>

Prepends values to a list key (preserving their order at the front of the list) or prepends string values to a scalar key. Prohibited on hash keys.

#pop <key [<count_or_subkeys...>]>

Pops elements from a list key by count (defaulting to 1), or deletes specified subkeys from a hash key. The count must be a non-negative integer (0 pops nothing); anything else halts with E_TYPE_MISMATCH.

#target_section <basename [in <namespace>]>

Redirects output to the specified section, pushing it onto the target stack.

#end_section

Pops the target stack and restores the enclosing output target (see "Output").

#create_section <basename [in <namespace>]>

Clears or creates a section buffer without redirecting output.

#emb[<options] <recipe> [<arg_sections...>]>

Inserts an unresolved embed placeholder/promise into the current section. Options in [...] accept comma-separated key-value pairs (e.g. aim=..., in=..., with_blinder=..., callbacks=...). (callbacks=... [Planned] {???}: the bundle is switched during the expansion, but assembly calls no callbacks yet, so it has no visible effect.) Bare #emb recipe acts as a direct section transclusion.

The recipe may be qualified (NS:recipe, or in=NS). An unqualified recipe is resolved late, during assembly: from the lens vertex if the promise sets aim=, otherwise starting from the namespace in which the enclosing section was requested (which may be a child of the namespace where the section's text was found).

Operands after recipe are bound to $ARGV0, $ARGV1, etc. in the recipe body during assembly. They are bound when the promise is expanded, in the context of the caller: NS:name starts its resolution at NS (lenses are ignored); an unqualified name starts at the caller's namespace; $ARGVn passes on the caller's own n-th operand. As a stopgap, when the promise switches the lens (aim=...), every operand must be tagged: old=name resolves as seen from the caller, new=name from the new lens vertex; an untagged operand on such a promise halts with E_UNTAGGED_OPERAND. (This may change once the semantic of operands is settled.)

#emb[aim=as_TOC] toc_entry old=sec1 new=sec7

#set_parents <namespace <parents...>>

Declares inheritance edges in the DAG. Validates cycle freedom (halting with E_CYCLIC_INHERITANCE). Parents need not exist yet; see E_UNDEFINED_PARENT.

#start_group [flavor] / #end_group [flavor]

Scoped configuration management and rollback boundaries. Supports optional bracketed options #start_group[callbacks=...]. Configuration changes made by these options are always undone at #end_group, whatever the flavor restores: by an undo action at priority 1000.0 (see group__postpone()), i.e. after the configuration is restored at priority 0.0 and after the undo actions with lower priorities. An unknown bundle name halts with E_UNKNOWN_BUNDLE before the group is opened.

#include [how] <data>

Slurps and processes input. For non-container modes (filename, string), strictly takes exactly 1 target argument. For container modes (section, configkey), strictly requires position 2 as the joiner key and position 3 as the target. Arity violations halt with E_SYNTAX_ERROR.

#include_scoped [how] [joiner] <data <flavor>>

Wraps inclusion in a group named flavor, guaranteeing complete rollback of configuration modifications upon EOF.

## [comment]

Comment line recognized by _default_comment_recognizer. Fuses preceding and succeeding text chunks across the comment boundary.

PRIMITIVE API REFERENCE

These are the low-level methods on the Text::Fab object that user-defined Parsers can call. The user-facing directives (like #set) are defined by the Parser, whose role is to convert these directives into calls to the Text::Fab methods listed below.

Output Control Primitives

out__target_section( $basename, $namespace ): Redirects subsequent output to the specified section, pushing it onto the target stack (see "Output"). Creates the section if it does not exist. The directive #target_section calls it.
out__end_section(): Pops the current target from the target stack, restoring the enclosing one. On an empty stack, and on the base entry of postponed output, it does nothing. The directive #end_section calls it.
out__flush(): Prints the postponed output, the section of the base entry of postponed output (_main:body), assembled, to Fab/output_handle, and empties that section. Does nothing if output was never postponed. See "Output".
out__create_section( $basename, $namespace ): Creates a section, clearing its content if it already exists.
out__embed( $recipe_basename, $recipe_ns, \@arg_sections, \%options ): Places an embed placeholder/promise into the current target section. \@arg_sections are positional operands mapped to $ARGVn. The %options hash can contain features like aim, with_blinder, or callbacks (the last one [Planned] {???}, see #emb). An undefined $recipe_ns is kept as such: the recipe is resolved late, during assembly, as described for #emb; so are the operands.
out__get_joined( $spec, \@joiners, \%options ): A utility for converting a section (representable as a list of text-parts and embed-promises) into a single string by joining text chunks and filtering embeds.

Configuration Primitives

cfg__set( $key, @values ): Sets the value of a key. For a scalar key, accepts at most one value (empty @values sets undef, exactly one value sets the scalar). Passing more than one value halts with E_TYPE_MISMATCH. For a list key, @values sets the list elements (or accepts a single array reference). For a hash key, @values accepts an even-sized list of key-value pairs or a single hash reference; an odd number of items halts with E_TYPE_MISMATCH.
cfg__append( $key, @values ): Appends one or more items to a list key, appends a string value to a scalar key, or merges an even-sized list of key-value pairs (or a hash reference) into a hash key. Appending an odd number of items to a hash key halts with E_TYPE_MISMATCH.
cfg__prepend( $key, @values ): Prepends one or more items to a list key (preserving their order at the front of the list), or prepends a string value to a scalar key. Calling cfg__prepend on a hash key is prohibited and halts with E_TYPE_MISMATCH.
cfg__get( $key, [$offset] ): When $offset is omitted, returns the scalar value, array reference, or hash reference. When $offset is provided, acts as an array index for list keys, or as a sub-key for hash keys. Passing $offset on a scalar key halts with E_TYPE_MISMATCH.
cfg__get_joined( $key, \@joiners, \%options ): A powerful utility that retrieves a list-type or hash-type key and joins its elements into a single string. The @joiners array provides one or more separators that are cycled through when joining. In particular, for hashes, joiner 0 is between the first key and value, the second separates the first and the second key/value pair etc.; hence with a single joiner, it is also used inside and between pairs. The %options can control complex formatting, for example specify permitted numbers of joiners modulo scalar @joiners. (E.g., if “joiners come in pairs”, then the number of emitted joiners must be even!) This is the option permitted_joiners_modulo: a list of the permitted remainders (or one remainder), each in 0 .. scalar(@joiners)-1; for the pairs above, permitted_joiners_modulo => [0]. So a hash is joined as the flat list of its keys and values, key1, value1, key2, value2, ...; with joiners ('=', '&', ';') the hash (a=>1, b=>2, c=>3) gives a=1&b;2=c&3. For a hash the rule counts all these joiners, i.e. 2n-1 for n pairs (so with two joiners, [0] never holds for a hash). A violation halts with E_TYPE_MISMATCH.
cfg__pop( $key, [@args] ): Pops count elements from a list key (defaulting to 1), or deletes one or more subkeys from a hash value when keys are passed. For a list key, count must be a non-negative integer (0 pops nothing); otherwise halts with E_TYPE_MISMATCH. Popping more elements than the list holds halts with E_LONG_POP.
cfg__prepend_elt( $key, $offset, $value ): Inserts $value at array index $offset for list keys, or inserts/updates a key-value pair [$k, $v] for hash keys. (The parser for directive may implement only the flavors with $offset in 0 and -1 only, e.g., “as if” in cfg__prepend_last( $key, $value ) etc. {???})

Working with Hashes

In addition to scalars and lists, the framework supports hashes as a first-class configuration data type. This is controlled by two special configuration keys:

Fab/hash_keys

Analogous to Fab/list_keys, this is a hash whose keys are the names of configuration variables that should be treated as hashes by the primitive operations.

# In a config file (registering a hash name):
#append Fab/hash_keys my_parameters 1

Fab/hash_key_sort_order

This key holds a subroutine reference that defines how the keys of a hash should be sorted when the hash needs to be treated as an ordered list (e.g., for cfg__get_joined). The subroutine receives two keys as arguments. The default is standard string comparison.

# In Perl code, to set numeric sorting:
$fab->cfg__set('Fab/hash_key_sort_order', sub { $_[0] <=> $_[1] });

Fab/max_recursion_depth

An integer setting the maximum permitted depth of the active embed stack during assemble(). Defaults to 1000. If recursive embeds exceed this ceiling, expansion halts with an E_CIRCULAR_EMBED exception.

Fab/max_include_depth

An integer setting the maximum number of nested include__text() calls (#include and its variants) while the input is read. Defaults to 200. One more halts with E_INCLUDE_TOO_DEEP.

Grouping and Postponement Primitives

group__start( [$flavor_name, \%options] )

Starts a scoped group. Which configuration keys group__end() restores depends on the flavor (full or partial undo); see #start_group in "Undoable Configuration Changes"" in "Grouping: Scoped.

group__end( [$flavor_name] )

Ends a scoped group.

group__postpone( $depth, $priority, $how, $method_name, @args )

Schedules a Text::Fab method call (or code reference) to run when the group at relative depth $depth is closed. All postponed actions have strict undo semantics: failure in one action does not abort the remaining undo actions or snapshot rollback. $priority is a real number: Priorities < 0 execute PRE-undo while the group scope is still active. Priority 0.0 is the snapshot rollback point where configuration is restored. Priorities >= 0 execute POST-undo in the restored enclosing scope. $how must be 'append' or 'prepend' to define execution order among actions with equal priority.

The closing frame remains active on the stack during unwinding, so actions can target the closing group at depth 0 or parent groups at depth > 0. Insertions into existing current/future priority buckets are consumed directly by the active loop without recalculation. Insertions into the past (time-machine actions) or into non-existing priorities trigger on-demand recalculation of the priority array.

(The parser for directives may allow more user-friendly UI, such as specifying “a path” to this group, as in “the last-but-one x group before the last two y groups.)

priority_for_file_close( [$dist_from_end] )

Calculates file-closing priority based on distance from the end: distance 0 (last closed file) is -1000.0, distance 999 is -1999.0, and distance >= 1000 is -2000.0 (overflow).

Input and Inclusion Primitives

include__text( $how, [$joiner_key], $source, [\%opts] )

$fab->include__text('filename',  $path,               [\%opts]);
$fab->include__text('string',    $content,            [\%opts]);
$fab->include__text('section',   $joiner_key, $spec,  [\%opts]);
$fab->include__text('configkey', $joiner_key, $key,   [\%opts]);

Slurps or expands content according to $how. An optional trailing hash reference \%opts is accepted across all modes. It holds the joining options, the same as the %options of cfg__get_joined() (such as permitted_joiners_modulo), and is passed on to the joining step in the 'section' and 'configkey' modes; in the 'filename' and 'string' modes it is accepted and ignored. The directive #include[...] passes its bracketed options here.

Processes included input. If $how is 'filename', $data is read from disk and processed. The files being included are mirrored in the configuration key opened_includes. A file may include itself, directly or through other files: whether this ends depends on what the file does (e.g. on a guard set in the configuration, or on which callbacks are active when it is read), which Fab cannot know. So Fab does not forbid this. Instead, it limits inclusions of all kinds (files, strings, sections, configuration keys) are nested at most Fab/max_include_depth deep (default 200); one more halts with E_INCLUDE_TOO_DEEP, whose message names the file open most often. An unreadable file halts with E_FILE_READ_ERROR. If $how is 'string', $data is processed directly as an in-memory string.

If $how is 'section' or 'configkey', requires $joiner_key (which is ignored when reading from a scalar-valued key); the $source is interpreted as a section-name or a key, correspondingly. This retrieves the list of joiner from configuration and joins the content using cyclic separators before processing.

Namespace and Inheritance Primitives

NS__set_parents( $namespace, @parents ): Registers the list of immediate parent namespaces for $namespace in the inheritance DAG (stored under Fab/inheritance_graph/$namespace/parents). Before writing, the inheritance graph is validated for cycles; adding a parent that would create a directed cycle halts with E_CYCLIC_INHERITANCE.
assemble( $root_spec ): Assembles and expands the section specified by $root_spec (e.g. Namespace:Basename or bare Basename defaulting to namespace _main). Coordinates lens affection massaging, positional argument binding ($ARGV0...), and blinder masks.

Configuration Prefix and Call Stack API

For recursive processing, the Configuration contains a stack of prefixes at Fab/call_stack_prefixes. Primitives are provided to safely interact with keys relative to this stack.

(We list only the most basic retrieval primitives; the rest follows the same principle.)

call_stack__push( $prefix, $filter_name ): Pushes a new configuration prefix onto the call stack at Fab/call_stack_prefixes, and $filter_name onto Fab/call_stack_filter_name. The name is required (otherwise E_TYPE_MISMATCH; an empty name counts as none), so the two lists stay paired: the root prefix 'Fab/' has no name, and the i-th name belongs to the (i+1)-th prefix.
call_stack__pop(): Pops the topmost prefix from Fab/call_stack_prefixes and the name pushed with it from Fab/call_stack_filter_name. Guards the root prefix 'Fab/' from being popped, halting with E_UPLEVEL_TOO_DEEP on underflow.
cfg__get_prefixed( $key ): Retrieves the value of $key using only the current, most recent prefix (i.e., CURRENT_PREFIX/$key).
cfg__get_prefixed_scan( $key ): Retrieves the value of $key by searching for it under each prefix in the call stack, from most recent to least recent.
cfg__get_uplevel_scan( $level, $key ): Retrieves the value of $key by first popping $level prefixes from the call stack and then performing a prefixed search.
cfg__set_prefixed( $key, @values ): Sets the value of $key using the current (most recent) prefix.

For setting things “uplevel on this stack”, there are two methods matching two types of access:

cfg__set_prefixed_uplevel( $level, $key, @values ): As cfg__set_prefixed(), but goes back $level steps on the stack of prefixes. This matches retrieval via cfg__get_prefixed().
cfg__set_prefixed_uplevel_fixup( $level, $key, @values ): As cfg__set_prefixed_uplevel(), but first runs “fixup”: copying the preceding value at $level to the levels between this one and the current level — stopping when value is found on a particular level. This matches retrieval via cfg__get_prefixed(): this call won’t change the results of these calls on the levels between $level and the current one.

ADVANCED TOPICS

Input Processing and Recursion

The primitive to support the #include directive is include__text($how, @args). If $how is filename, the file $data is searched for and slurped; if not, $data is the input string to be processed. A special form #include_scoped ensures that configuration changes made by the included file are temporary by wrapping the call in a group named by the last argument. Active inclusion paths are mirrored in opened_includes.

Recursive processing (e.g., using one Text::Fab “process” to filter data for another) is managed entirely within the Configuration hash. A stack of configuration prefixes is maintained in Fab/call_stack_prefixes (which usually starts as just Fab/). When a named recursive filter is invoked, its name is pushed to the filter name stack Fab/call_stack_filter_name. This also pushes a certain prefix onto Fab/call_stack_prefixes via call_stack__push(). Since Text::Fab looks up its own settings under the prefixes on this stack, the most recent one first (see "The Configuration Hash" for the list of these settings), this allows a special predefined configuration (e.g., Fab/myList2ScalarFilter) to take precedence without conflicting with the “normal” operation of the Fab.

The uplevel family of configuration directives [Planned] {???} operates with respect to both the group stack and this call stack, allowing for powerful, controlled communication between different layers of processing. (The default parser has no such directives yet; currently only the primitives exist: cfg__get_uplevel_scan(), cfg__set_prefixed_uplevel(), cfg__set_prefixed_uplevel_fixup() and group__postpone().)

Namespace Resolution and Inheritance of output sections

The inheritance system is a layer of logic built on top of the primitive operations. It is controlled entirely by the Configuration hash. Here we illustrate the API by how the corresponding directives may look like. The API has an extra prefix NS__, and omits the UI verbiage.

#set_parents <namespace> <parent1> <parent2> ...: A user-facing directive that calls cfg__set on a key within the Configuration that stores the inheritance DAG. For example, it might set Fab/inheritance_graph/MyNamespace/parents to [Parent1, Parent2].

Resolution order. A section NS:name is looked up in NS first, then in the parents of NS depth-first, in the order in which they were declared; a namespace reached a second time (through another path of the DAG) is not searched again. In other words, a parent A is treated as a mix of A proper with whatever defaults A overrides, and this whole mix comes before the next parent. So with Page having parents (A, B), and both A and B having the parent Base, the search order is Page, A, Base, B: a section defined both in Base and in B is taken from Base. (A breadth-first order, Page, A, B, Base, may later be offered as an option. {???})

Parents may be declared before they exist. If the search reaches a parent which has neither sections nor parents of its own, assembly halts with E_UNDEFINED_PARENT.

The Assembly Phase uses this graph to resolve embedded placeholders/promises in the output sections; below we denote them as #emb. The resolution can be controlled with extra data:

#emb[aim=<Aim>, ...] <recipe> [<arg_sections...>]

This performs lens-guided resolution. The requested aim is passed through _massage_lens across the active lens stack via static joining rules and registered affections in Fab/lens_affects, establishing the virtual root in the DAG. When the virtual root is not the root, this makes the name resolution "blind" to any “override sections” up the tree/DAG. (Recall that the root contains the most specific overrides.)

The resolved operands are accessible as $ARGV0, $ARGV1, etc. in the recipe body. Since such a promise switches the lens, every operand must be tagged old=... or new=... (see #emb in "DEFAULT (dbg0) DIRECTIVES REFERENCE"). {???}

#emb[with_blinder=<SubName>, ...] <recipe> ...

Here SubName provides a completely customizable analog of the previous call, looked up as a subroutine reference under Fab/blinders/$SubName. Instead of (or in addition to) obscuring the part of DAG which is not reachable from the aim, the output of this user-defined subroutine obscures an arbitrary subset of the DAG. Blinding a namespace masks that namespace's own section definitions from satisfying the embed, while allowing inheritance traversal to continue through its parents in the DAG. It receives the entire current embedding stack (each entry is the currently executed embedding promise) together with the start namespace, the used blinder name, and the output of the blinder for each level. It returns a dynamic list of namespaces to "blind" for this specific resolution and any sub-resolutions (which may happen during processing of this embedded section). A name not registered under Fab/blinders/ halts with E_UNKNOWN_BLINDER.

Embed Cycle Detection and Recursion [NEEDED ???]

During assembly, Text::Fab detects circular embeds along the active resolution branch. For unparameterized sections, re-entering the same namespace:basename triggers an E_CIRCULAR_EMBED exception.

For parameterized recipes (invoked with argument sections), re-entry is permitted provided the active argument frame differs. This enables recursive recipe decomposition and iterative processing, bounded by Fab/max_recursion_depth.

The argument frame consists of the bound operands: each operand is turned into the namespace:basename where its resolution starts when the promise is expanded, and an operand $ARGVn is aliased to the caller's own n-th operand. So a recipe may pass its aliased operands on (with $ARGV3 becoming $ARGV0 inside inner:

#target_section outer in L
#emb inner $ARGV3
#end_section

Support of Control-Flow-Like constructs (conditionals and/or loops)

Fab knows no conditional or loop directive itself: a user-defined directive parser recognizes them and calls the control API below. Fab keeps a control stack of the open constructs ("control blocks"), drops the input of a skipped branch, and moves the reading position for loops.

The control stack. Every input text has its own control stack: the top-level input (given to process_string() or include__text()), and each file, string, section or configuration key included in it by include__text() (e.g. by #include). It starts empty, and the stream must leave it empty (else E_UNCLOSED_CONTROL_BLOCK); the stack of the enclosing stream is restored after it. So a block cannot begin in one file and end in another. The stack is not part of the configuration, and groups do not restore it. The configuration key Fab/control__loop_indices lists the indices (0 is the bottom) of its loop-like frames; it is kept current, also after #end_group (as opened_includes). A frame (one open block) records:

the type given to control__start() (e.g. if, while);
the label: a frame with a defined label ('' is fine) is loop-like;
the skipping state: 0 live, 1 skipping, 2 skipping on entry: skipped whatever the parser asks (the block started inside a skipped part, or lies above a frame searching for a pointer, see control__jump_to());
the pointers: named positions in the input stream (for a loop: where a pass starts, where the loop ends, ...);
the loop counter, the invalidation pointer, and the name of a pointer being searched for (see control__jump_to()).

{???} Keeping this stack outside the configuration is provisional: where the state that #end_group must not restore should live (in the configuration, or in a separate store which only the API changes, as the group and include stacks are now) is an architectural question still to be deliberated carefully.

Skipping. The skipping state of the top frame (0 if the stack is empty) is the skipping state of the input. While it is not 0, Fab does not call the chunk preprocessor: the text is dropped. The interleaver and the directive parser get the state as an extra argument (see PLACEHOLDERS). In a skipped part the parser must still recognize the control directives, to keep track of the nesting, and should execute no other directive; the default (dbg0) parser executes no directive there.

Loops. A loop reads a part of the input several times: at its end, the parser jumps back with control__jump_to() to a pointer at its start. Each reading of the part is a pass; all the passes of one loop, from its start to its end, form an instance (a loop nested in another one starts a new instance in every pass of the outer loop). Text read again in the current instance of the innermost loop-like frame is a replay, and the parser gets a replay flag. The pointers of an instance are set in its first pass: in a replay, control__set_pointer() keeps a pointer already set (new names are still added).

The API. All the methods except control__skipping(), control__reentering() and control__loop_counter() must be called by the directive parser while it executes a directive (else E_CONTROL_OUTSIDE_DIRECTIVE); positions refer to that directive. control__set_skipping(), control__end(), control__set_pointer() and control__define_invalidation_pointer() work on the top frame, and halt with E_DANGLING_CONTROL_END if the stack is empty.

control__start( $type, $is_skipping, [$label] )

Pushes a frame for the construct whose start directive is being executed. Its skipping state is 2 if the input is skipped, else 1 if $is_skipping is true, else 0. With a defined $label, the frame is loop-like.

The exception is re-entering: the directive being executed re-enters a frame when it was reached by a control__jump_to() to that frame's own start position (so it is the frame's start directive, read again; see control__reentering()). Then the first call made during this directive whose $type and $label equal those of the re-entered frame (labels both undefined, or equal) pushes nothing: it continues that frame. It sets the frame's skipping state again, 1 if $is_skipping is true, else 0 (a frame skipped on entry stays so), e.g. after the loop condition is evaluated again; the frame keeps its pointers, loop counter and invalidation pointer. Any other call pushes a new frame, as usual. Only the frame jumped to is re-entered, even if other frames start at the same position.

control__set_skipping( 'flip' | $boolean )

Sets the skipping state of the top frame; ignored if the state is 2 or a pointer is being searched for.

control__end( $expected_type )

Pops the top frame. Halts with E_MISMATCHED_CONTROL_END if its type differs, and with E_POINTER_NOT_FOUND if it searches for a pointer which this directive does not set.

control__set_pointer( $name, [$before], [$force] )

Records the pointer $name in the top frame. With $before true, the pointer is the position where the directive being executed begins, and it is recorded at once. Otherwise the pointer is the position where the text of this directive ends: the offset the directive parser returns (a jump made by the directive does not change it). Fab learns that position only from that return value, so this pointer is recorded after the parser returns. A forward search (see control__jump_to()) for this pointer therefore ends only after this directive: while the parser is still busy with the directive, the frame is still skipping, and control__skipping() still returns a true value (the #endwhile of the example below relies on this). In a replay, an existing pointer is kept unless $force is true.

control__define_invalidation_pointer( $expected_type )

Records the position after this directive as the first position not in the top frame (halts with E_MISMATCHED_CONTROL_END if its type differs). When reading reaches this position, the frame ends, as by control__end(). A loop typically sets it at its end directive, before it jumps back.

control__jump_to( $pointer_name, [$block_label], [$increment] )

Continues reading at a pointer of a frame: of the frame labelled $block_label, or, without a label, of the innermost loop-like frame (of the top frame if no frame is loop-like; halts with E_BLOCK_LABEL_NOT_FOUND if the stack is empty). Ignored in a skipped part.

If no open frame has the label $block_label, Fab searches forward for a block with this label which starts directly inside the top frame (or at the top level of the input text, if the stack is empty): the input is skipped until such a block starts; blocks with this label nested deeper are skipped too. The block found is the target frame, and Fab searches for the pointer in it, as below. If the top frame ends first (or the input text, if the stack was empty), Fab halts with E_BLOCK_LABEL_NOT_FOUND.

If the pointer is set, the frames above the target frame end, its loop counter is increased by $increment (default 0), and reading continues at the pointer. If the pointer is strictly before the position where the start directive of the frame begins, or at or after its invalidation pointer, the target frame ends too. (A pointer set with $before at the start directive is not before it: the jump reads the start directive again, see control__start().)

If the pointer is not set yet, Fab searches forward: the target frame skips until control__set_pointer() sets the pointer in it; then it is live again, and its counter is increased by $increment. If the frame ends first (by control__end(), or when its invalidation pointer is reached), Fab halts with E_POINTER_NOT_FOUND, unless the directive that ends it also sets the pointer (after itself): this is how a break in the first pass leaves a loop.

The frames above the target frame stay open during the search, but their skipping state becomes 2, so that a directive like #else inside them cannot make them live, and their own end directives, which the search reads, still end them. Example (with the directives of t/19-control-flow.t; O, I are labels):

#while[O] outer         frame O
#while[I] inner         frame I, above O
#if inner               frame IF, above I
#break[O]               jump to "end" of O: not known in the first pass, so a search
#endif                  ends IF
#endwhile               ends I (it sets "end" of I, not of O)
#endwhile               ends O; it also sets "end" of O, so no E_POINTER_NOT_FOUND
after                   read live

If the break ended I and IF at once, the search would meet #endif with O on top (E_MISMATCHED_CONTROL_END); and even without that check, the #endwhile of I would set the pointer end of O, ending the search at the wrong place. When the pointer is known (e.g. in a later pass), the frames above the target end at once, since reading continues at the pointer and their end directives are not read.

Loop limits. The configuration key Fab/control__loop_limits (a list, default [1e300]) limits the jumps back in one instance of a frame: entry D (counting from 0) is the most jumps back for a frame at loop depth D, the number of loop-like frames below it (in this input and in the inputs that include it); the last entry counts for all greater depths. A jump to a pointer before or at the position where the directive being executed begins is a jump back. One jump back beyond the limit halts with E_LOOP_LIMIT. (A loop that tests its condition at its start directive, read again, jumps back once per live pass, so a limit of L allows L live passes; a loop that tests at its end directive jumps back once fewer.)

control__skipping()

The skipping state of the input (of the top frame; 0 if the stack is empty). The parser may need it after control__set_pointer() with $before, which may end a search at once.

control__reentering()

Undefined, except while the directive parser executes a directive which re-enters a frame; then a hash reference { type => ..., label => ... } naming that frame.

A directive re-enters a frame when control__jump_to() jumped to a pointer of that frame which is the frame's own start position (the position where its start directive begins; e.g. a pointer set with $before by the start directive), and the frame stays open; the next directive executed, which begins there, re-enters it. That directive is the frame's start directive, read again: it should continue the frame, not start a new one. Fab does not guess which of the directive's calls of control__start() continues the frame: it is the first one with the type and the label of the frame, and the parser learns from control__reentering() which frame it is.

This matters when one directive starts several frames. Example: #grid[R,C] 2 3 starts a loop over 2 rows (type grid, label R) and, inside it, a loop over 3 columns (type grid, label C). Both start at the position of #grid, and the parser sets a pointer top in each, before the directive. At #endgrid, the parser jumps to top of C while columns remain, else to top of R (which ends C, the frame above R); after the last cell, it ends C and R (control__end('grid') twice):

#endgrid:  if    (control__loop_counter('C') + 1 < 3) { control__jump_to('top', 'C', 1) }
           elsif (control__loop_counter('R') + 1 < 2) { control__jump_to('top', 'R', 1) }
           else  { control__end('grid'); control__end('grid') }

reading of #grid   control__reentering()    the parser calls control__start() for
1st                undefined                R and C (and sets "top" in each)
2nd, 3rd           { label => 'C', ... }    C only: C continues (counter 1, then 2)
4th                { label => 'R', ... }    R (continues, counter 1), then C: a new C
                                            (counter 0; the jump to R had ended the old one)
5th, 6th           { label => 'C', ... }    C only

That gives 6 cells, in 2 rows of 3. (t/19-control-flow.t has this parser.) For a directive which starts one frame, like #while, the parser need not ask: it calls control__start() as on the first reading, and the frame is continued; its control__set_pointer('top', 1) is then in a replay, and keeps the pointer.

control__loop_counter( [$label] )

The loop counter of the loop-like frame labelled $label, or (without a label) of the innermost loop-like frame: the sum of the increments given to control__jump_to() in this instance. Halts with E_BLOCK_LABEL_NOT_FOUND if there is no such frame.

Example: a directive parser may implement #while COND ... #endwhile and #break like this (t/19-control-flow.t has a complete parser with conditionals and loops):

#while:     control__start('while', $skipping || !COND, '');
            control__set_pointer('top', 1);       # before #while: it is read again
#endwhile:  control__set_pointer('end');          # after #endwhile
            if (control__skipping()) { control__end('while') }
            else { control__define_invalidation_pointer('while');
                   control__jump_to('top', undef, 1) }
#break:     control__jump_to('end') unless $skipping;

In a live pass, #endwhile jumps back to #while, which evaluates COND again; when COND is false, the body is skipped, and #endwhile ends the frame. A #break in the first pass searches forward for end, which #endwhile sets; in later passes end is known and is the invalidation pointer, so the jump ends the loop.

Sometimes a callback may need a computationally expensive combination of individual entries from a configuration hash. (On may think of a regular expression depending on several values in this hash.) To avoid recalculating it every time, the callback may use the logic like this:

unless ($cache_at_count == (my $new_cnt = $fab->{config}{cnt__parser_REx})) {
  $cache = my_recalc_cache($Fab);
  $cache_at_count = $new_cnt;
}

This assumes that $fab->{config}{cnt__parser_REx} changes every time one of the variables used by my_recalc_cache() is updated (or restored). To support this, one register each of these variables with the “dependent counter” cnt__parser_REx using the API:

config__append_dependent_counter($counter_key, $KEY1, ...) [Planned] {???}

(Warning: the implementation is free to change the counter back when a group is undone if it knows all the relevant-and-changed variables were properly undone. So (theoretically) one cannot rely on the counter to increase-on-modification.

A tacit assumption — caveat usor

Summary: Before starting a design using Text::Fab, check that inside your preprocessor, the internal representation of the partially processed input can be easily enough kept in the Configuration Hash. We assume that this is going to hold as far as the preprocessor “proceeds with a well-defined processing pipeline”.

---

In many cases one needs to allow the configuration driving the logic of work of a preprocessor to be changeable by a directive at any time. (For example, a text in one programming language may inline a text in another programming language!)

The design of Fab’s API can make this (seem to be?) hard: (due it its up→down approach to the search of directives) the directive above would interrupt the input to the preprocessor; — but sometimes the preprocessor may need a non-local view on the input to decide how to deal with it. “Just seeing what comes before the directive”, then “Just seeing what comes after the directive” (in a separate invocation!) may be not enough for such a non-local view.

On the other hand, one should be ready for the situation when the input after this directive should be handled differently than the input before this directive. So if the target domain requires handling such situations, then any implementation of such a preprocessor must be ready to deal with such hiccups when processing the input (either by handling the update of its config, or bulking out with an error message). While doing this with a bottom→up design may seem easier, the experience shows that often it is prohibitively hard to implement. (E.g., when nested constructions are dealt with by recursive calls, the calls to reconfigure_me() would happen deep inside the stack of recursive calls!)

In short: to implement such a design through Text::Fab the only possible complication is: how to preserve the internal representation of the “preceding not-yet-fully-digested input” within the preprocessor “when the preprocessor is reentered” after handling such a directive?

With the bottom→up approach (when the preprocessor may “call some API” to handle the directive) this is easy: it is easy to make sure that such a call may update “the configuration” of the preprocessor while keeping “the internal representation of the already-read-input-to-preprocess” fully intact. However, when the preprocessor is controlled from a Fab, what happens instead of “calling the API” is: “exit the preprocessor”, then somebody else (the Fab!) calling the API, then “reenter the preprocessor again”. Therefore when the preprocessor can see that the chunk it obtained is a part of “some larger construct” (so it is not ready to be flush()'ed to the output section yet), it must preserve its internal state “somewhere”, then read it back when it is reentered (if it is ever reentered). This is the price to pay for (a lot of) work offloaded to Fab.

CONCLUSION:

In such situations the preprocessor needs a guarantee that it is going to be reentered;
Its internal data should be easily translatable to/from formats supported by the configuration hash.

(Indeed, this hash is the only supported storage provided by Fab’s API.)

In other words: the preprocessor cannot be stateless now, but as far as the state may be (de)serialized easily, it is not a big deal. So one possible tacit assumption about designs using Text::Fab may be:

The preprocessing is hierarchical, each level of hierarchy of not-yet-fully-processed-input may be put into a value suitable for the Configuration Hash, and the steps remaining to finish processing these data are not affected by the directive above.

————————————————————————————————————

Next, inspect how not-stateless but still “easy to implement” preprocessors may look like.

Example B: We have a text which may contain nested “blocks”, each block made of one or more paragraphs. The nesting is defined by the indent level (as in Python). The “class” of each block is determined by inspecting the immediate start of the its first paragraph, as well as inspecting the immediate end of the last paragraph of block (provided this last paragraph is “on the same indent level as the start” — so it is not in a nested subblock).

In addition to this we allow #define directives which take a whole line — but are not considered as interrupting the logic of indentation-which-defines-nested-depth.

Implementation: To implement “reenterancy”, the stored data consists of the stack of already-preprocessed text of non-yet-finished blocks. When preprocessing a chunk of input finishes a block, the preprocessor can inspect its start and end, determine its class, pop()'s the known “intermediate textural representation” from the stack, and converts it to the final representation in the way suitable for this class. When this was the outermost block, the result is put into the output section. Otherwise it is appended to the preceding entry on the stack (the containing block).

————————————————————————————————————

Example H: As above, but the rules for massaging a block depend not only on its class, but also on the classes of its parents. This requires “the most general” hierarchical model of the pipeline of not-yet-fully-processed input.

In this case again the handled data may be flush()'ed to an output section only when the outermost encountered block is terminated. Until then, one should maintain the stack of “live blocks” (not yet terminated), as well as a forest of “dead” (already terminated, but waiting the finite processing) blocks. Each terminated block has its class and the list of content, elements are either “a literal text” (preprocessed-except-the-final-step), or another (nested) terminated sub-block. The live blocks are likewise, but the class is not yet fully known, and the last nested sub-block may be live.

How to preserve this in the Configuration Hash? We have a planar rooted tree with vertices assigned the class (or undef for live vertices), and leaves containing “a plain text”. When flush()'ing to an output section, we essentially scan depth-first through the vertices, maintaining the stack of classes (they correspond to the path from the root to the current vertex). So the only bookkeeping data in addition to the stack of classes is the (parallel) stack of “where the currently-not-yet-finished sub-block ends”. (So we can pop() the stack of classes when this end is reached.)

So if we store the leaves in the depth-first order, we need to interlace this list with markers “start sub-block of the class CLASS ending at position POS” (or introduce nested start and end markers). To design an easy representation, suppose that the semantic of “massaging the blocks” allows introdution of empty "plain text leaves" between every two adjacent sub-vertices; then we may assume that “end-subblock” entries are represented by out-of-bound elements (e.g., undef), and the plain text leaves are alternating with “start subblock” entries in the stored list, as in flattened list of the following form:

0thLlvTxt1
  Blk1
    1stLvl1Txt11
      Blk11
        2ndLvl1Txt111
      undef
    1stLvl1Txt12
      Blk12
        2ndLvl1Txt121
      undef
    1stLvl1Txt13
  undef
0thLlvTxt2
  Blk2
    1stLvl1Txt21
      Blk21
        2ndLvl1Txt211
      undef
    1stLvl1Txt22
      Blk22
        2ndLvl1Txt221
      undef
    1stLvl1Txt23
  undef
0thLlvTxt3

Observe alternation Txt/BlkClass/Txt/BlkClass/Txt/… after every undef. It is trivial to process such a list when flush()'ing. When filling the list, one should additionally maintain a stack of positions in this list where live block starts, so one can change a placeholder-for-its-class to the calculated class when the end-of-block is found. (Other than this, all the results of pre-processing the input land at the end of the list.)

————————————————————————————————————

Example A: Consider a macroprocessor for macros with arguments; assume that the argument-separators and argument-terminators for a macro on a certain level of nesting may be contained in the output of nested-deeper macros. So one needs to expand the deeper macros before the handling of enclosing macros is finished. Can one support directives occuring deep inside the nested arguments — but we assume they cannot be contained in the macro-expansions?

Here the data may be even easier than in the preceding example. Since the output produced up to the start of the outermost no-yet-terminated macro may be flash()'ed, the stack of unprocessed data may contain the ID of the outermost no-yet-terminated macro, the list of its completed arguments, then the already expanded prefix of the current argument, then the ID of the next no-yet-terminated macro, the list of its completed arguments etc. In addition to this list, it is enough to have the list of offsets at which the IDs of the macros live.

When a terminator of a macro is found, one pops from the list above the macro ID and the completed arguments, and performs the macrosubstitution. There are two ways to proceed with this string: it is either appended to “the already expanded prefix of the current argument” (as above) of the enclosed macro (if the expansion should not be macro-re-expanded) — and optionally scanned for macro-argument-delimiter or -terminator, or it is prepended to the buffer containing the not-yet-processed input (otherwise). The desired choice is determined by the semantic of the macro-expansion.

————————————————————————————————————

Example E: As above, but the directives may appear as the result of macro-expansion.

This case is much more involved, since it seems it is too late for the Fab to inspect the internal state of the macro-processor. So the preprocessor should scan its output itself: it has the full access to the De-Interlacer/Parser/etc callbacks designed to detect the directives.

WARNING: However, the directives are not designed to work with not-yet-completed buffers. For best result, De-Interlacer should return “the high-water mark”: the offset before which the directives cannot appear — so there is no sense to rescan this part when the buffer is extended.

ERROR REFERENCE

The preprocessor will halt with one of the following error codes if a fatal condition is met.

Parsing and Input Errors (`E_Input`)

E_SYNTAX_ERROR: A directive does not conform to the grammar, or an unrecognized directive is encountered.
E_FILE_READ_ERROR: An #included file cannot be opened or read.
E_INCLUDE_TOO_DEEP: Inclusions (#include, include__text()) are nested deeper than Fab/max_include_depth.

Grouping Errors (`E_Group`)

E_MISMATCHED_END_GROUP: An #end_group <flavor> does not match the currently open group.
E_DANGLING_END_GROUP: An #end_group is found with no matching #start_group.
E_INVALID_GROUP_NESTING: An attempt to nest groups in a way forbidden by the Configuration.
E_UPLEVEL_TOO_DEEP: An uplevel primitive (or, [Planned] {???}, an uplevel directive), a call stack pop, or group__postpone() targets a level that does not exist.

Configuration Errors (`E_Config`)

E_TYPE_MISMATCH: A list-specific operation is attempted on a scalar key, or vice-versa. Also used for wrong numbers or kinds of arguments to primitives (for example an odd number of items for a hash key, an invalid cfg__pop count), and when a join violates permitted_joiners_modulo.
E_LONG_POP: Popping too many elements.

Namespace and Assembly Errors (`E_Assembly`)

E_CYCLIC_INHERITANCE: An #set_parents directive creates a loop in the inheritance DAG.
E_UNDEFINED_PARENT: During assembly, the resolution reaches a parent namespace (declared with #set_parents) which has neither sections nor parents of its own. (#set_parents itself accepts parents which do not exist yet.)
E_ROOT_NOT_FOUND [Planned]: A section specified via #set_root [Planned] does not exist. (Currently assemble() takes the root section as its argument.)
E_NO_ROOTS_SPECIFIED [Planned]: The assembly phase is triggered, but no roots were ever defined with #set_root [Planned].
E_EMBED_NOT_FOUND: A section referenced by an #emb placeholder/promise cannot be resolved.
E_PROMISE_IN_DIRECT_OUTPUT: An #emb is met while the output goes directly to Fab/output_handle, and Fab/output_postpone is error (see "Output").
E_CIRCULAR_EMBED: An #emb chain results in a loop during the final assembly.
E_UNKNOWN_BUNDLE: A requested callback bundle is not registered under Fab/callbacks_bundle/.
E_ARGV_EMPTY: Referenced an $ARGVn section parameter outside of a recipe invocation.
E_ARGV_BOUNDS: Referenced an $ARGVn index that exceeds the number of supplied section arguments.
E_UNKNOWN_BLINDER: A promise names a blinder (with_blinder=...) which is not registered under Fab/blinders/.
E_UNTAGGED_OPERAND: A promise that switches the lens (aim=...) has an operand not tagged old= or new=.

Control Flow Errors (`E_Control`)

E_MISMATCHED_CONTROL_END: control__end() (e.g. for #endif) or control__define_invalidation_pointer() names a type other than that of the open control block (e.g. a #for loop).
E_DANGLING_CONTROL_END: An end directive (or another call of the control API that needs an open block) was met when the control stack of the input stream was empty.
E_UNCLOSED_CONTROL_BLOCK: The end of an input stream (a file, a string, a section or a key being included, or the top-level input) was reached while one or more control blocks were still open.
E_POINTER_NOT_FOUND: A forward search started by control__jump_to() reached the end of the target control block without the pointer being set.
E_BLOCK_LABEL_NOT_FOUND: control__jump_to() searched forward for a block with the given label, and the block enclosing the jump (or the input text) ended first; or control__loop_counter() found no open loop-like block with the given label; also when they are called without a label and no suitable block is open. The message names the pointer and the label.
E_LOOP_LIMIT: A jump back would exceed the limit in Fab/control__loop_limits for the depth of the loop.
E_CONTROL_OUTSIDE_DIRECTIVE: A method of the control API that needs the position of a directive was called while no directive was being executed.

PLACEHOLDERS

Callback signatures, as the code calls them now. {???}

The active callbacks are the hash in Fab/callbacks (or, if it holds a name, the bundle Fab/callbacks_bundle/NAME); the default bundle is dbg0. They are fetched once at the start of each process_string().

interleaver( $buffer, $offset, $skipping, $is_replay ): Returns the offset of the next directive at or after $offset, or undef if there is none. (It does not get $fab.) $skipping is the skipping state of the input (0: live) and $is_replay the replay flag (see "or loops)"" in "Support of Control-Flow-Like constructs (conditionals and). dbg0: a # at the start of a line.
comment_recognizer( $fab, $buffer, $offset ): Called with the offset found by the interleaver (also in a skipped part of the input). Returns the offset after the comment if the directive there is a comment, else undef. The text before a comment is not flushed; it is fused with the text after it. dbg0: a line starting with ##.
directive_parser( $fab, $buffer, $offset, $skipping, $is_replay ): Executes the directive at $offset (usually by calling primitive methods) and returns the offset where scanning resumes (unless the directive jumps, see control__jump_to()). $skipping and $is_replay are as for the interleaver. dbg0: the directives in "DEFAULT (dbg0) DIRECTIVES REFERENCE"; in a skipped part of the input it executes none.
chunk_preprocessor( $fab, $chunk ): Returns the processed text of a chunk (the text between directives, with comments fused); Text::Fab appends it to the current target section. dbg0: returns $chunk unchanged.
error_handler( $fab, $type, \%details ): Called by dbg0's parser with $type 'unknown_directive' (%details: directive, line) or 'malformed_directive' (line). dbg0: dies with E_SYNTAX_ERROR.
Blinders: Fab/blinders/NAME->( $fab, \%info ): %info has embed_stack (the frames of the enclosing promises, each with section, options and blinder_output), start_namespace, blinder_name and target_section. Returns the list of namespaces to blind.

The text below is the earlier plan for these sections. {???}

Future sections will detail the full APIs and default implementations for:

The Parser: How it interacts with the main loop and calls primitives.
Chunk Preprocessors: The interface for the "stomach".
Blinder Subroutines: The data structures passed to them.
The default configuration schema under the Fab/ key:

Fab/callbacks_bundle/dbg0

A hash reference containing the canonical operational engine callbacks: interleaver, directive_parser, chunk_preprocessor, comment_recognizer, and error_handler. {??? The active callbacks bundle is referenced via Fab/callbacks.}

The comment_recognizer callback receives ($fab, $buffer, $offset) and returns the offset after the comment line if recognized (defaulting to lines prefixed with ##), which triggers chunk fusion across the comment boundary. {??? Docs of the rest?}

EXPORT

None by default.

TODO

Maybe use caching counters to automate conversion of `callbacks` from a string to a hash of subroutines???

Need a test which merges peano with infix, then one implementing all of 50-54 basing only on unary and Peano primitives. m4???

Document which parts of "Fab/" Config are accessed through an explicit path, and which through a dynamic one.

A conflict between installable components's names may arise; so they need to allow redefinition of the name/prefix they use.  Or:
they requestion "Foo", get back "Foo12" (so the requested name should not end in a digit).

Efficient replacement for FreezeThaw. Handle DAG config - but probably not an arbitrary graph with weak links inside??? Handle a tree and the knowledge of where is the gluing of branches???

How switching the filter interacts with dynamic switching of Fabs??? Where the switched filter stores its data: explicitly in subsets of config??? Interaction with dynamic parts of the config??? Compartmentalization of config???

How to indicate resolution in the caller/callee namespace??? What is the default now???

How to decide “executed vs. postponed” for promises: which promises are expanded now, and which "pass through" unchanged? Compare a “do-when” “watermark” set in the promise and in the piping stage?

Traceback of provenance of errors in presence of promises??? Traceback of who created the promise in its metadata???

User-configurable logging/debugging output???

For postponed actions need an extra argument "ignore errors": with 2 would just put into eval, with 1 converted to a warning, negative values stop the remaining undo actions for that many frames (with -2 stopping for the current frame and one more).

One needs to prohibit __get()ing a reference. Use __getref() when you expect a reference (but no checks for reference are performed).

MISS: include__text() reading from a named filehandle (e.g. STDIN), not only from a file name or a string.

MISS: detecting the end of a job. A job should be a group (the outermost one); ending the job closes all open groups down to it, running everything postponed (which may add output, promises or new groups) until nothing is left, and printing the postponed output is one of these postponed actions. There should be ways to declare that every job consists of one top-level include, or that a particular (top-level only???) include finishes the input when it ends.

MISS: (eventually) an engine for “streamlining to STDOUT”: if a promise keeps the output from continuing (the output after it is postponed, see "Output"), and somebody with authority claims that the promise is finished and ready to ship, the (interactive???) output may be continued.

“This authority” is self-declared (just a unique name), but may be overridable by other entities.

MISS: input__end($n) with $n==0 ending the current input stream, or also $n enclosing streams.

MISS: (found by t/54_m4.t) a way to put text in front of the rest of the current input, read as one text with it, e.g. input__prepend($text): an m4 macro expansion may begin a word or a macro call that ends in the text after it. include__text(string => ...) reads a separate stream, so it cannot do this.

MISS: (found by t/54_m4.t and t/55_cpp.t) output of text without the chunk preprocessor, e.g. out__text($text), for text that is already processed: m4's undivert (copying a diversion into the output) and cpp's #pragma (passed through unexpanded).

MISS: need to specify what happens with a group on input__end(): survives / should be closed before / close-if-still-open.

MISS: input__end_to($group_type): end input of the files opened up to the the opened-latest-group of the given type.

Likewise for “closing enclosed groups”??? Join together into 1 API by introducing suitable groups???

  May run forever??? Even the language if group__start, group_end, group__postpone (with the arguments to the last command
  restricted to these 3 calls) is probably Turing complete???

MISS: _-prefixed versions of config__*() which check the prefix _ __ ___ of the key and make it read/write only.

MISS: pipelining preprocessor with “filters”: filters consists of 3 components: preprocessor, massaging of promises, and the concatenator of the resulting candidates-for-sections.

Uplevel w.r.t. the recursion stack should be recalculated into the usual uplevels; a special do-nothing group may be used.

allow not 1 active preprocessor engine, but a stack of them, chained, and have a cfg__pop() primitive method.

Macros are templates marked as OK-to-be-ignored.

Hash to mark macros as “types”, with per-type-warning configurable? Which way to hash, types to macros, or macros to types, or pairs???

An extra parameter $is_replay for the syntax callbacks for the non-first pass. (Probably, the boundary of the preceding pass cannot be part of this??? DONE: the interleaver and the directive parser get it.

Hash values. Setting/getting them employs a serialization method, by default JSON???

     !!!!!!!!!!!!!!!!!!!!!!!!!!!!	THIS IS TOO DANGEROUS.
					Access control to Fab/* keys should be guarded by ungettable/settable types of values ???

#unset — but maybe it is going to make undo harder???

Recursive invocation is needed for massaging data between different levels (currently, these are convertable to output sections):

• The hash
• Input file names
• Input strings
• Output sections
• Arguments to directives/promises

A top→down approach to recognizing the “reconfiguration directives”: but cannot we support a bottom→up approach too, when the preprocessor recognizes interlaced directives (<b>x<i>y</b>z</i>? How to deal with recursion: return an indicator that the state changed? But it is changing permanently, one needs to filter out changes — but how the internal calls know the filters needed for the enclosing calls???

Resolving to _main namespace is broken (too early); it should be left undefined, so the resolver knows it should do the late resolution — at runtime, when the lenses may be set up differently. DONE: unqualified recipes and operands are now resolved during assembly.

the preprocessor needs a guarantee that it is going to be reentered

---

The default mode of processing is equivalent to writing to a buffer __default__, and at the end executing the file

= = = = = filterbuffer(raw) __default__

Doing the same abstraction once more, the buffer with that contents may be also created dynamically, then loaded via filterbuffer(). On the gripping hand, this may be also

= = = = = filterbuffer(raw) __default__pre
= = = = = filterbuffer(raw) __default__
= = = = = filterbuffer(raw) __default__post

With (initially) empty buffers __default__pre and __default__post (to which one has a chance to write later, during creation of __default__!). And what one writes to these buffers may be triplicated likewise. Etc.

AUTHOR

Ilya Zakharevich, <ilyaz@cpan.com>

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.22.2 or, at your option, any later version of Perl 5 you may have available.

To install Text::Fab, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Text::Fab

CPAN shell

perl -MCPAN -e shell
install Text::Fab

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Preview of architecture

Example of usage

Massaging of lens names

THE PROCESSING MODEL

The Configuration Hash

The Output Sections

Target Section Stacking and Scoping

Output

Grouping: Scoped/Undoable Configuration Changes

DEFAULT (dbg0) DIRECTIVES REFERENCE

PRIMITIVE API REFERENCE

Output Control Primitives

Configuration Primitives

Working with Hashes

Grouping and Postponement Primitives

Input and Inclusion Primitives

Namespace and Inheritance Primitives

Configuration Prefix and Call Stack API

ADVANCED TOPICS

Input Processing and Recursion

Namespace Resolution and Inheritance of output sections

Embed Cycle Detection and Recursion [NEEDED ???]

Support of Control-Flow-Like constructs (conditionals and/or loops)

Support of Caching configuration-related data [Planned] {???}

A tacit assumption — caveat usor

ERROR REFERENCE

Parsing and Input Errors (E_Input)

Grouping Errors (E_Group)

Configuration Errors (E_Config)

Namespace and Assembly Errors (E_Assembly)

Control Flow Errors (E_Control)

PLACEHOLDERS

EXPORT

TODO

SEE ALSO

AUTHOR

COPYRIGHT AND LICENSE

Module Install Instructions

Keyboard Shortcuts

DEFAULT (`dbg0`) DIRECTIVES REFERENCE

Parsing and Input Errors (`E_Input`)

Grouping Errors (`E_Group`)

Configuration Errors (`E_Config`)

Namespace and Assembly Errors (`E_Assembly`)

Control Flow Errors (`E_Control`)