NAME

Pod::Literate - Write literate programming within POD, for general applications or specifications generated by AI, Copilot, Claude, ChatGPT.

VERSION

This specification describes

< Literate Version >=
our $VERSION='0.0.3';

DESCRIPTION

Literate programming naturally structures code based on its written specification. Traditionally this permitted organization of very large applications in languages that initially focused on single-file development, which slowly extended to multiple files used as basic imports. With the advance of projects spanning many directories and isolation into objects, and the propensity to author code with minimal comments and documentation, literate programming became an historical curiosity.

With the advent of AI tooling focused on software development assistance, it is possible to leverage literate programming to build specifications for applications and interfaces. Given the natural-language focus of this approach, LLMs benefit from and can better maintain consistency with specifications. Code again becomes focused on small, isolated, well-tested goals that can be merged together to form larger systems.

These specifications can be processed in two ways. A tangle command permits coalescing all named sections into the final application. A weave operation generates the specification will all relevant visible elements. In Pod::Literate, the weave application if perlpod itself and its associated formatters (pod2man etc.). Only a specialized tangle command is provided to use the specification to generate the target application.

Backwards Incompatibility

IMPORTANT: This is an early version. Monitor the version change logs for additional information. Nothing is expected to "explode", but there may be fundamental changes.

Section declaration versus addition

The warning/error behavior of literate Name versus literate +Name may change, be dependent on commandline flags, etc.

Hidden sections

When weave is provided, both of begin and for sections will become visible. There may be some benefit to having hidden sections that still generate code.

Unused sections

Unused sections may become failures, warnings, etc.

Commands

Commands are specified using the augmented format identifier literate:command.

Weave

Weave options and overall behaviors are subject to change.

FORMAT FEATURES

The module, Pod::Literate, is implemented using the formatting features of perlpod. There are two modes that are determined by the attributes of the command paragraph, either =begin (and its associated =end command), or =for. Both declare named sections of code, but they behave differently during a weave operation.

< Set the mode >=
if($target eq 'literate:command') { $$self{$cache}{mode}='command' }
else { $$self{$cache}{mode}=$$attribute{'~really'} }

begin and end

The begin and end command paragraphs are used to create sections that are hidden during a weave operation, and must appear in pairs:

=begin literate Descriptive name

...

=end literate

When processing the start of these blocks, a title attribute is set with the "Descriptive name" which is used to identify the target section into which the text contents will be stored:

< Set target from begin >=
if($$attribute{title}) { my $s=$$attribute{title}; $s=~s/^\+//; $self->targetSet($s) }
$self->initCollection();

Notice, at this time, that the classic additive syntax for a named section is supported, in particular

=begin literate +Descriptive name

...

=end literate

but this is equivalent to the non-additive name. In particular, if sections appear more than once, the contents are always concatenated.

< Populate the target >=
$self->sectionAppend($name, $self->collected());

for

The =for command paragraph denotes sections that are not hidden during a weave operation. The ending of a for section uses a period:

=for literate Descriptive name

...

=for literate .

At this time the period is mandatory and signals the end of content collection:

< Conclude for section >=
$self->clearCollection()

The for paragraph behaves slightly differently from begin/end. In the case of =for Content, there are no attributes on the element and the "Content" is loaded in the plain text handler. That means the section target name is actually configured when the end of the for element is reached.

< Set target from for >=
my $s=$collected; $s=~s/^\+//;
$self->targetSet($s);
$self->initCollection();

Section names

Sections named in =begin and =for tags are limited only by perlpod functionality. Each name is referenced in code by placing it within commented identifiers as:

#< Descriptive name >#

Leading and trailing spaces are ignored, but otherwise only the sequence ># would be illegal in a name:

< The section name regexp >=
my $tagL='#<';
my $tagR='>#';
my $reSection=qr/$tagL\s*(.+?)\s*$tagR/;

TANGLE

The tangle operation first validates the sections, then builds them in topological order, the saves the sections to the output file.

< Tangle Operation >=
sub tangle {
	my ($self)=@_;
	my (%tobuild,@todo); # prebuild to keep the loops separate
	$self->prepareSections();
	#< Topologic sort for sections >#
	foreach my $name (@todo) { $self->tangleSection($name) }
	#< Write the output file >#
	return $self;
}

Building a single section requires replacing the section name references with the dependent code. As a first pass, if the reference appears on a single line with an indent, that indentation is copied into all lines of the referenced code before insertion. Otherwise the code is inserted verbatim.

< Tangle a single section >=
sub tangleSection {
	my ($self,$name)=@_;
	my $section=$$self{$cache}{section}{$name};
	foreach my $import (keys %{$$self{$cache}{dep}{$name}}) {
		my $re=sectionRegexp($import);
		my $code=$$self{$cache}{section}{$import}; $code=~s/\n+$//s;
		while($section=~/^(\s*)$re\s*$/m) {
			my $icode=indent($code,$1);
			$section=~s/^\s*$re\s*$/$icode/em;
		}
		$section=~s/$re/$code/ge;
	}
	$$self{$cache}{section}{$name}=$section;
}

After the POD has been fully scanned and the end of the document is reached, the above method is all that's required to tangle the document.

< Tangle the document >=
$self->tangle();

The module contains a number of supporting methods for the tangle operation.

< Tangle Methods >=
#< Tangle Operation                  >#
#< Tangle a single section           >#
#< Finding all section references    >#
#< A method to validate the sections >#

Preparing the sections

Target construction depends on sections that are defined and that can be merged together successively into larger pieces. Cyclic dependencies between sections are not permitted. Missing sections are, obviously, an issue, and sections that are never used are a warning as they suggest potential typos (the latter is not yet provided).

< A method to validate the sections >=
sub prepareSections {
	my ($self)=@_;
	my ($changed,$sections,%dep,%missing,%cyclic)=(1);
	$sections=$$self{$cache}{section};
	#< Populate the dependency tree >#
	if(%missing) { die 'Name referenced but not defined:  '.join(q{; },sort keys %missing) }
	#< Compute the dependency closure >#
	if(%cyclic) { die 'Sections contain cyclic dependencies:  '.join(q{, },sort keys %cyclic) }
	$$self{$cache}{dep}=\%dep;
	return $self;
}

The dependency tree follows the straightforward construction, identifies any missing names, and builds a classic, hash-focused graph of dependency{parent}{child} relationships.

< Populate the dependency tree >=
while(my ($k,$code)=each %$sections) {
	foreach my $name (codeSectionNames($code)) {
		if(!defined($$sections{$name})) { $missing{$name}=1 }
		$dep{$k}{$name}=1;
	}
}

After the dependency graph has been constructed, the full closure is needed to ensure there are no cyclic dependencies and to facilitate the topological sort. A regular, slow closure is computed.

< Compute the dependency closure >=
while($changed) { $changed=0;
	foreach my $ka (keys %dep) {
	foreach my $kb (keys %{$dep{$ka}}) {
		foreach my $kc (keys %{$dep{$kb}}) {
			my $d=$dep{$ka}{$kb}+$dep{$kb}{$kc};
			if(!defined($dep{$ka}{$kc})||($dep{$ka}{$kc}<$d)) {
				$dep{$ka}{$kc}=$d; $changed=1;
				if($ka eq $kc) { $cyclic{$ka}=1 }
			} }
	} }
}

Topological sorting of the dependency graph uses a small trick in Perl: Since there are no cycles in the graph, there must be at least one section name having zero dependencies, which take(s) priority during per-section tangling. After those sections have been removed, the remaining sections to build must again have at least one with no dependencies, and so forth. Slicing permits removing the zero-dependency sections as the order is built.

< Topologic sort for sections >=
while(my ($ka,$refb)=each %{$$self{$cache}{dep}}) { $tobuild{$ka}={map {$_=>1} keys(%$refb)} }
while(my @names=sort grep {!%{$tobuild{$_}}} keys(%tobuild)) {
	push @todo,@names;
	delete(@tobuild{@names});
	foreach my $name (values %tobuild) { delete(@$name{@names}) }
}

Building the output file

The output filename is chosen by specifying the target filename in a =for command paragraph.

< Handle tangle target >=
if(($mode eq 'command')&&($collected=~/^target=(.+)$/)) {
	$$self{$cache}{ofn}=$1;
	$self->clearCollection();
}
elsif(($mode eq '=for')&&($collected=~/^target:(.+)$/)) {
	print STDERR "=for literate target: is deprecated; use =for literate:command\n";
	$$self{$cache}{ofn}=$1;
	$self->clearCollection();
}

As the output filename is itself a named section, creating the output file is a simple matter of writing that section.

< Write the output file >=
if(my $ofn=$$self{$cache}{ofn}) {
if(my $oft=$self->section($ofn)) {
	open(my $fh,'>',$ofn) or die "open: $!";
	print $fh $oft;
	close($fh);
} }

MODULE

The perlpod format is provided in a basic module, Pod::Literate:

< Literate.pm >=
package Pod::Literate;
use parent 'Pod::Simple';
#< Literate Version >#
use strict;
use warnings;
#< Name the cache          >#
#< The section name regexp >#
#< Constructor             >#
#< Accessors and Setters   >#
#< POD Handlers            >#
#< Tangle Methods          >#
1;
__END__

POD Methods

A subclass of Pod::Simple must implement handlers for the start and end of command paragraphs, specifically =begin, =end, and =for, as well as a mechanism to gather text. When an element starts, its attributes identify if it matches this module. Because POD enforces begin/end balancing, the ending element has no similar attribute.

< POD Handlers >=
sub _handle_element_start {
	my ($self,$element,$attribute)=@_;
	my $target=$$attribute{target}//'';
	#< Fail for non-data paragraphs >#
	if($target!~/^literate(?::command)?$/) { return }
	#< Handle a start element >#
}

sub _handle_element_end {
	my ($self,$element,$attribute)=@_;
	if($element eq 'for') {
		my ($mode,$collected)=@{$$self{$cache}}{qw/mode collector/};
		$collected=~s/\n$//s;
		#< Handle tangle target  >#
		#< Handle an end element >#
	}
	elsif($element eq 'Document') {
		#< Tangle the document >#
	}
}

sub _handle_text {
	my ($self,$text)=@_;
	#< Handle a text element >#
}

For elements

The behavior of the =for tags is dependent on whether or not the target has been set. If a =for literate Name has already appeared, and the interim text already scanned, then the =for literate . will trigger a begin event. That must store the collected text, then reset collection to grab the period character associated with this paragraph.

< Handle a start element >=
if(my $name=$self->target()) {
	#< Populate the target >#
	$self->clearCollection()->initCollection();
	return;
}

After the period is grabbed, the end event will need to clear the collector.

< Handle an end element >=
elsif(($mode eq '=for')&&($collected eq '.')) {
	$self->clearCollection();
}

When an end event fires after =for literate Name, however, the collected text, not an attribute, is used to populate the target.

< Handle an end element >=
elsif($mode eq '=for') {
	#< Set target from for >#
}

Begin/End elements

A true =begin element still looks like a basic for element, though it does have a title provided as an attribute.

< Handle a start element >=
if($element eq 'for') {
	#< Set the mode          >#
	#< Set target from begin >#
}

When the end element is reached, the section can be populated.

< Handle an end element >=
elsif($self->target()) {
	$self->sectionAppend($self->target(), $self->collected());
	$self->clearCollection();
}

Text collection

The only task to perform when basic text is read is to populate the collector.

< Handle a text element >=
if(defined($$self{$cache}{collector})) { $$self{$cache}{collector}.=$text."\n" }

Data paragraphs

At this time, only data paragraphs are supported in =begin and =for blocks. A non-data paragraph, that is a "plain paragraph", may contain normal POD commands and additional plain documentation. Only data/code paragraphs are supported at this point. In the future, the :literate type might be used to support embedding actual POD in the result, but this requires additional parsing/handling not yet available. If it is encountered, fail:

< Fail for non-data paragraphs >=
if($target eq ':literate') { die 'Non-data paragraphs are not currently supported via ":literate"' }

Instance construction

Each instance of a Pod::Literate is a Pod::Simple object with the addition of the internal cache used during scanning.

< Constructor >=
sub new {
	my ($class)=@_;
	my $self=$class->SUPER::new();
	#< Initialize the cache >#
	return $self;
}

Accessors and Setters

Each instance of the module must maintain an internal collection of structures. At this time, the cache is stored in a single instance variable with an unlikely name:

< Name the cache >=
my $cache='--PODLITERATECACHE--';

It is checked during initialization just in case:

< Initialize the cache >=
if(exists($$self{$cache})) { die 'Basic Pod::Simple object unexpectedly contains the Pod::Literate cache key' }
$$self{$cache}={
	ofn      =>{},    # the output filename for the tangled application
	section  =>{},    # the collection of named sections
	mode     =>'',    # scanning mode, to distinguish begin/for tags
	target   =>undef, # defined and destroyed while scanning
	collector=>undef, # defined and destroyed while scanning
	#
	dep      =>{},    # used during tangling to compute dependencies
};

Each instance of the module will have the following functionality as described below:

< Accessors and Setters >=
#< A method to set the target                  >#
#< A method to get the target                  >#
#< A method to append to sections              >#
#< A method to retrieve sections               >#
#< A regular expression for a specific section >#
#< A method to initialize collection           >#
#< A method to format collected text           >#
#< A method to clear collection                >#
#< A method to remove leading indents          >#
#< A method to indent code                     >#

Targets

The target section name may appear with a starting command, or may be part of the scanned text. Therefore, it may need to be set at different times when scanning the document.

< A method to set the target >=
sub targetSet {
	my ($self,$name)=@_;
	$$self{$cache}{target}=$name;
	return $self;
}

Whereas most uses of the target name will be to select that named section, it is also used in a boolean context while scanning. Having a general way to get the target name is useful.

< A method to get the target >=
sub target {
	my ($self)=@_;
	return $$self{$cache}{target};
}

Sections

For each named section, scanned text can be appended (concatenated).

< A method to append to sections >=
sub sectionAppend {
	my ($self,$section,$text)=@_;
	$$self{$cache}{section}{$section}.=$text;
}

During tangle operations, the code within the sections will be retrieved to populate the application or other sections.

< A method to retrieve sections >=
sub section {
	my ($self,$section)=@_;
	return $$self{$cache}{section}{$section};
}

Also during tangle operations, it is important to exactly match section references within code. This method constructs the regular expression used for those replacements.

< A regular expression for a specific section >=
sub sectionRegexp {
	my ($name)=@_;
	return qr/$tagL\s*\Q$name\E\s*$tagR/;
}

That regular expression is also used to find references within code text.

< Finding all section references >=
sub codeSectionNames {
	my ($code)=@_;
	my %names;
	while($code=~m/$reSection/g) { $names{$1}=1 }
	return keys(%names);
}

Text collection

Text is collected in an instance variable

< A method to initialize collection >=
sub initCollection {
	my ($self,$text)=@_;
	$$self{$cache}{collector}=$text//'';
	return $self;
}

The collected text can be retrieved in a form suitable for inclusion in a section reference.

< A method to format collected text >=
sub collected {
	my ($self)=@_;
	return unindent($$self{$cache}{collector});
}

At the end of of each section being scanned, the target section name and the collection mechanism are cleared.

< A method to clear collection >=
sub clearCollection {
	my ($self)=@_;
	delete(@{$$self{$cache}}{qw/target collector/});
	return $self;
}

Formatters

The first line of a section defines the indent level.

< A method to remove leading indents >=
sub unindent {
	my ($s)=@_;
	if($s=~/^(\s+)/) {
		my $indent=$1;
		$s=~s/^\Q$indent\E//mg;
	}
	return $s;
}

If a code reference is indented, the inserted code is indented by the same amount.

< A method to indent code >=
sub indent {
	my ($code,$indent)=@_;
	if(!$indent) { return $code }
	$code=~s/^/$indent/emg;
	return $code;
}

TODO

The following are being considered. No particular priority order here.

todo Weave command is required if the begin/for names are supposed to show up in the POD

The names are clear within the specification, but the generated POD doesn't contain the names. Presumably the generated pod would show something like

< Descriptive name > +=
Content
todo Support =for literate import:filename

Documents may appear in sections, and one large POD can become difficult to manage. With a specialized weave command, it would be sensible to support imports.

todo Handle multiple output filenames

The target=filename can simmply set the output and all subsequent sections will be collected therein. If another filename is encountered, the scanner will start putting sections there. With an import statement, the filenames would be fairly well scoped, and perhaps import should explicitly scope any set filename.

done First line indentation is removed, to match indentation at the point of reference
done First line indentation applies to every line in the reference
done Inline references

A reference within a line is handled correctly. if(#< ref >#) expands properly if ref is a single line, with no additional spaces. If ref is a multiline, it's still inserted at that spot, with its inner newlines, but with no terminal newline.

todo Inline declaration

Support for X<name>code to declare single lines of code within a paragraph.

todo Detect/reject invalid sequences like begin/for, begin/begin, for title/for title
todo Investigate colon-prefixed =for :literate command for the command strings.
todo Blank code lines

Blank lines before/after subroutines, for example, are not straightforward to specify. Inside =for blocks, those would violate podchecker which expects no whitespace-only lines. It may be this is entirely the responsibility of a later style tool, but it should be possible to get actual blank lines without having to put those blank lines around the reference. Maybe some #nl or equivalent, which would still look nicer than some lengthy # blank line # type of comment.

Another approach would be, since this is Perl POD, to permit enabling spaces around balanced blocks that are denoted, such as sub.

todo Cross reference

Lexicographic index of all section names etc. (Page numbers seem irrelevant, is it just 1.2.3.4 style subsection numbering?). Possibly leverage `makeindex`, if available.

todo Module pod

(how to denote pod that belongs with modules?)

todo Tests

Generally tests will appear in different sections, but it's possible to write a specification where the tests are associated with each section. Unfortunately, for concatenated sections the tests might be more difficult to arrange prior to tangling.

todo Reference wildcards

Support #< Name.* ># to select all matching names, alphabetical order