NAME
Pod::Literate - Write literate programming within POD, for general applications or specifications generated by AI, Copilot, Claude, ChatGPT.
VERSION
This specification describes
our $VERSION='0.0.2';
DESCRIPTION
Literate programming naturally structures code based on its written specification. Traditionally this permitted organization of very large applications in languages that initially focused on single-file development, which slowly extended to multiple files used as basic imports. With the advance of projects spanning many directories and isolation into objects, and the propensity to author code with minimal comments and documentation, literate programming became an historical curiosity.
With the advent of AI tooling focused on software development assistance, it is possible to leverage literate programming to build specifications for applications and interfaces. Given the natural-language focus of this approach, LLMs benefit from and can better maintain consistency with specifications. Code again becomes focused on small, isolated, well-tested goals that can be merged together to form larger systems.
These specifications can be processed in two ways. A tangle command permits coalescing all named sections into the final application. A weave operation generates the specification will all relevant visible elements. In Pod::Literate, the weave application if perlpod itself and its associated formatters (pod2man etc.). Only a specialized tangle command is provided to use the specification to generate the target application.
Backwards Incompatibility
IMPORTANT: This is an early version. Monitor the version change logs for additional information. Nothing is expected to "explode", but there may be fundamental changes.
- Section declaration versus addition
-
The warning/error behavior of
literate Nameversusliterate +Namemay change, be dependent on commandline flags, etc. - Hidden sections
-
When weave is provided, both of
beginandforsections will become visible. There may be some benefit to having hidden sections that still generate code. - Unused sections
-
Unused sections may become failures, warnings, etc.
- Commands
-
Commands like
literate target:namemight change. Considering:literatefor these. - Weave
-
Weave options and overall behaviors are subject to change.
FORMAT FEATURES
The module, Pod::Literate, is implemented using the formatting features of perlpod. There are two modes that are determined by the attributes of the command paragraph, either =begin (and its associated =end command), or =for. Both declare named sections of code, but they behave differently during a weave operation.
$$self{$cache}{mode}=$$attribute{'~really'};
begin and end
The begin and end command paragraphs are used to create sections that are hidden during a weave operation, and must appear in pairs:
=begin literate Descriptive name
...
=end literate
When processing the start of these blocks, a title attribute is set with the "Descriptive name" which is used to identify the target section into which the text contents will be stored:
if($$attribute{title}) { $self->targetSet($$attribute{title}=~s/^\+//r) }
$self->initCollection();
Notice, at this time, that the classic additive syntax for a named section is supported, in particular
=begin literate +Descriptive name
...
=end literate
but this is equivalent to the non-additive name. In particular, if sections appear more than once, the contents are always concatenated.
$self->sectionAppend($name, $self->collected());
for
The =for command paragraph denotes sections that are not hidden during a weave operation. The ending of a for section uses a period:
=for literate Descriptive name
...
=for literate .
At this time the period is mandatory and signals the end of content collection:
$self->clearCollection()
The for paragraph behaves slightly differently from begin/end. In the case of =for Content, there are no attributes on the element and the "Content" is loaded in the plain text handler. That means the section target name is actually configured when the end of the for element is reached.
$self->targetSet($collected=~s/^\+//r);
$self->initCollection();
Section names
Sections named in =begin and =for tags are limited only by perlpod functionality. Each name is referenced in code by placing it within commented identifiers as:
#< Descriptive name >#
Leading and trailing spaces are ignored, but otherwise only the sequence ># would be illegal in a name:
my $tagL='#<';
my $tagR='>#';
my $reSection=qr/$tagL\s*(.+?)\s*$tagR/;
TANGLE
The tangle operation first validates the sections, then builds them in topological order, the saves the sections to the output file.
sub tangle {
my ($self)=@_;
my (%tobuild,@todo); # prebuild to keep the loops separate
$self->prepareSections();
#< Topologic sort for sections >#
foreach my $name (@todo) { $self->tangleSection($name) }
#< Write the output file >#
return $self;
}
Building a single section requires replacing the section name references with the dependent code. As a first pass, if the reference appears on a single line with an indent, that indentation is copied into all lines of the referenced code before insertion. Otherwise the code is inserted verbatim.
sub tangleSection {
my ($self,$name)=@_;
my $section=$$self{$cache}{section}{$name};
foreach my $import (keys %{$$self{$cache}{dep}{$name}}) {
my $re=sectionRegexp($import);
my $code=$$self{$cache}{section}{$import}; $code=~s/\n+$//s;
while($section=~/^(\s*)$re\s*$/m) {
my $icode=indent($code,$1);
$section=~s/^\s*$re\s*$/$icode/em;
}
$section=~s/$re/$code/ge;
}
$$self{$cache}{section}{$name}=$section;
}
After the POD has been fully scanned and the end of the document is reached, the above method is all that's required to tangle the document.
$self->tangle();
The module contains a number of supporting methods for the tangle operation.
#< Tangle Operation >#
#< Tangle a single section >#
#< Finding all section references >#
#< A method to validate the sections >#
Preparing the sections
Target construction depends on sections that are defined and that can be merged together successively into larger pieces. Cyclic dependencies between sections are not permitted. Missing sections are, obviously, an issue, and sections that are never used are a warning as they suggest potential typos (the latter is not yet provided).
sub prepareSections {
my ($self)=@_;
my ($changed,$sections,%dep,%missing,%cyclic)=(1);
$sections=$$self{$cache}{section};
#< Populate the dependency tree >#
if(%missing) { die 'Name referenced but not defined: '.join(q{; },sort keys %missing) }
#< Compute the dependency closure >#
if(%cyclic) { die 'Sections contain cyclic dependencies: '.join(q{, },sort keys %cyclic) }
$$self{$cache}{dep}=\%dep;
return $self;
}
The dependency tree follows the straightforward construction, identifies any missing names, and builds a classic, hash-focused graph of dependency{parent}{child} relationships.
while(my ($k,$code)=each %$sections) {
foreach my $name (codeSectionNames($code)) {
if(!defined($$sections{$name})) { $missing{$name}=1 }
$dep{$k}{$name}=1;
}
}
After the dependency graph has been constructed, the full closure is needed to ensure there are no cyclic dependencies and to facilitate the topological sort. A regular, slow closure is computed.
while($changed) { $changed=0;
foreach my $ka (keys %dep) {
foreach my $kb (keys %{$dep{$ka}}) {
foreach my $kc (keys %{$dep{$kb}}) {
my $d=$dep{$ka}{$kb}+$dep{$kb}{$kc};
if(!defined($dep{$ka}{$kc})||($dep{$ka}{$kc}<$d)) {
$dep{$ka}{$kc}=$d; $changed=1;
if($ka eq $kc) { $cyclic{$ka}=1 }
} }
} }
}
Topological sorting of the dependency graph uses a small trick in Perl: Since there are no cycles in the graph, there must be at least one section name having zero dependencies, which take(s) priority during per-section tangling. After those sections have been removed, the remaining sections to build must again have at least one with no dependencies, and so forth. Slicing permits removing the zero-dependency sections as the order is built.
while(my ($ka,$refb)=each %{$$self{$cache}{dep}}) { $tobuild{$ka}={map {$_=>1} keys(%$refb)} }
while(my @names=sort grep {!%{$tobuild{$_}}} keys(%tobuild)) {
push @todo,@names;
delete(@tobuild{@names});
foreach my $name (values %tobuild) { delete(@$name{@names}) }
}
Building the output file
The output filename is chosen by specifying the target filename in a =for command paragraph.
if(($mode eq '=for')&&($collected=~/^target:(.+)$/)) {
$$self{$cache}{ofn}=$1;
$self->clearCollection();
}
As the output filename is itself a named section, creating the output file is a simple matter of writing that section.
if(my $ofn=$$self{$cache}{ofn}) {
if(my $oft=$self->section($ofn)) {
open(my $fh,'>',$ofn) or die "open: $!";
print $fh $oft;
close($fh);
} }
MODULE
The perlpod format is provided in a basic module, Pod::Literate:
package Pod::Literate;
use parent 'Pod::Simple';
#< Literate Version >#
use strict;
use warnings;
#< Name the cache >#
#< The section name regexp >#
#< Constructor >#
#< Accessors and Setters >#
#< POD Handlers >#
#< Tangle Methods >#
1;
__END__
POD Methods
A subclass of Pod::Simple must implement handlers for the start and end of command paragraphs, specifically =begin, =end, and =for, as well as a mechanism to gather text. When an element starts, its attributes identify if it matches this module. Because POD enforces begin/end balancing, the ending element has no similar attribute
For elements
The behavior of the =for tags is dependent on whether or not the target has been set. If a =for literate Name has already appeared, and the interim text already scanned, then the =for literate . will trigger a begin event. That must store the collected text, then reset collection to grab the period character associated with this paragraph.
if(my $name=$self->target()) {
#< Populate the target >#
$self->clearCollection()->initCollection();
return;
}
After the period is grabbed, the end event will need to clear the collector.
elsif(($mode eq '=for')&&($collected eq '.')) {
$self->clearCollection();
}
When an end event fires after =for literate Name, however, the collected text, not an attribute, is used to populate the target.
elsif($mode eq '=for') {
#< Set target from for >#
}
Begin/End elements
A true =begin element still looks like a basic for element, though it does have a title provided as an attribute.
if($element eq 'for') {
#< Set the mode >#
#< Set target from begin >#
}
When the end element is reached, the section can be populated.
elsif($self->target()) {
$self->sectionAppend($self->target(), $self->collected());
$self->clearCollection();
}
End elements
Text collection
The only task to perform when basic text is read is to populate the collector.
if(defined($$self{$cache}{collector})) { $$self{$cache}{collector}.=$text."\n" }
Instance construction
Each instance of a Pod::Literate is a Pod::Simple object with the addition of the internal cache used during scanning.
sub new {
my ($class)=@_;
my $self=$class->SUPER::new();
#< Initialize the cache >#
return $self;
}
Accessors and Setters
Each instance of the module must maintain an internal collection of structures. At this time, the cache is stored in a single instance variable with an unlikely name:
my $cache='--PODLITERATECACHE--';
It is checked during initialization just in case:
if(exists($$self{$cache})) { die 'Basic Pod::Simple object unexpectedly contains the Pod::Literate cache key' }
$$self{$cache}={
ofn =>{}, # the output filename for the tangled application
section =>{}, # the collection of named sections
mode =>'', # scanning mode, to distinguish begin/for tags
target =>undef, # defined and destroyed while scanning
collector=>undef, # defined and destroyed while scanning
#
dep =>{}, # used during tangling to compute dependencies
};
Each instance of the module will have the following functionality as described below:
#< A method to set the target >#
#< A method to get the target >#
#< A method to append to sections >#
#< A method to retrieve sections >#
#< A regular expression for a specific section >#
#< A method to initialize collection >#
#< A method to format collected text >#
#< A method to clear collection >#
#< A method to remove leading indents >#
#< A method to indent code >#
Targets
The target section name may appear with a starting command, or may be part of the scanned text. Therefore, it may need to be set at different times when scanning the document.
sub targetSet {
my ($self,$name)=@_;
$$self{$cache}{target}=$name;
return $self;
}
Whereas most uses of the target name will be to select that named section, it is also used in a boolean context while scanning. Having a general way to get the target name is useful.
sub target {
my ($self)=@_;
return $$self{$cache}{target};
}
Sections
For each named section, scanned text can be appended (concatenated).
sub sectionAppend {
my ($self,$section,$text)=@_;
$$self{$cache}{section}{$section}.=$text;
}
During tangle operations, the code within the sections will be retrieved to populate the application or other sections.
sub section {
my ($self,$section)=@_;
return $$self{$cache}{section}{$section};
}
Also during tangle operations, it is important to exactly match section references within code. This method constructs the regular expression used for those replacements.
sub sectionRegexp {
my ($name)=@_;
return qr/$tagL\s*\Q$name\E\s*$tagR/;
}
That regular expression is also used to find references within code text.
sub codeSectionNames {
my ($code)=@_;
my %names;
while($code=~m/$reSection/g) { $names{$1}=1 }
return keys(%names);
}
Text collection
Text is collected in an instance variable
sub initCollection {
my ($self,$text)=@_;
$$self{$cache}{collector}=$text//'';
return $self;
}
The collected text can be retrieved in a form suitable for inclusion in a section reference.
sub collected {
my ($self)=@_;
return unindent($$self{$cache}{collector});
}
At the end of of each section being scanned, the target section name and the collection mechanism are cleared.
sub clearCollection {
my ($self)=@_;
delete(@{$$self{$cache}}{qw/target collector/});
return $self;
}
Formatters
The first line of a section defines the indent level.
sub unindent {
my ($s)=@_;
if($s=~/^(\s+)/) {
my $indent=$1;
$s=~s/^\Q$indent\E//mg;
}
return $s;
}
If a code reference is indented, the inserted code is indented by the same amount.
sub indent {
my ($code,$indent)=@_;
if(!$indent) { return $code }
$code=~s/^/$indent/emg;
return $code;
}
TODO
The following are being considered. No particular priority order here.
- todo Weave command is required if the begin/for names are supposed to show up in the POD
-
The names are clear within the specification, but the generated POD doesn't contain the names. Presumably the generated pod would show something like
< Descriptive name > += Content - todo Support
=for literate import:filename -
Documents may appear in sections, and one large POD can become difficult to manage. With a specialized weave command, it would be sensible to support imports.
- todo Handle multiple output filenames
-
The
target:filenamecan simmply set the output and all subsequent sections will be collected therein. If another filename is encountered, the scanner will start putting sections there. With animportstatement, the filenames would be fairly well scoped, and perhapsimportshould explicitly scope any set filename. - done First line indentation is removed, to match indentation at the point of reference
- done First line indentation applies to every line in the reference
- todo Inline references
-
Most work has been focused on standalone references. What is the behavior of
{ named reference }when the reference is a single line? Multiline? - todo Inline declaration
-
Support for
X<name>codeto declare single lines of code within a paragraph. - todo Detect/reject invalid sequences like begin/for, begin/begin, for title/for title
- todo Investigate colon-prefixed
=for :literate commandfor the command strings. - todo Blank code lines
-
Blank lines before/after subroutines, for example, are not straightforward to specify. Inside
=forblocks, those would violate podchecker which expects no whitespace-only lines. It may be this is entirely the responsibility of a later style tool, but it should be possible to get actual blank lines without having to put those blank lines around the reference. Maybe some#nlor equivalent, which would still look nicer than some lengthy# blank line #type of comment.Another approach would be, since this is Perl POD, to permit enabling spaces around balanced blocks that are denoted, such as
sub. - todo Cross reference
-
Lexicographic index of all section names etc. (Page numbers seem irrelevant, is it just 1.2.3.4 style subsection numbering?). Possibly leverage `makeindex`, if available.
- todo Module pod
-
(how to denote pod that belongs with modules?)
- todo Tests
-
Generally tests will appear in different sections, but it's possible to write a specification where the tests are associated with each section. Unfortunately, for concatenated sections the tests might be more difficult to arrange prior to tangling.
- todo Reference wildcards
-
Support
#< Name.* >#to select all matching names, alphabetical order