NAME
IO::Compress::Lzop::FAQ -- Frequently Asked Questions about IO::Compress::Lzop
DESCRIPTION
Common questions answered.
Compatibility with Unix compress/uncompress.
This module is not compatible with Unix compress.
If you have the uncompress program available, you can use this to read compressed files
open F, "uncompress -c $filename |";
while (<F>)
{
...Alternatively, if you have the gunzip program available, you can use this to read compressed files
open F, "gunzip -c $filename |";
while (<F>)
{
...and this to write compress files, if you have the compress program available
open F, "| compress -c $filename ";
print F "data";
...
close F ;Accessing .tar.Z files
See previous FAQ item.
If the Archive::Tar module is instaled and either the uncompress or gunzip programs are available, you can use one of these workarounds to read .tar.Z files.
Firstly with uncompress
use strict;
use warnings;
use Archive::Tar;
open F, "uncompress -c $filename |";
my $tar = Archive::Tar->new(*F);
...and this with gunzip
use strict;
use warnings;
use Archive::Tar;
open F, "gunzip -c $filename |";
my $tar = Archive::Tar->new(*F);
...Similarly, if the compress program is available, you can use this to write a .tar.Z file
use strict;
use warnings;
use Archive::Tar;
use IO::File;
my $fh = new IO::File "| compress -c >$filename";
my $tar = Archive::Tar->new();
...
$tar->write($fh);
$fh->close ;Using InputLength to uncompress data embedded in a larger file/buffer.
A fairly common use-case is where compressed data is embedded in a larger file/buffer and you want to read both.
As an example consider the structure of a zip file. This is a well-defined file format that mixes both compressed and uncompressed sections of data in a single file.
For the purposes of this discussion you can think of a zip file as sequence of compressed data streams, each of which is prefixed by an uncompressed local header. The local header contains information about the compressed data stream, including the name of the compressed file and, in particular, the length of the compressed data stream.
To illustrate how to use InputLength here is a script that walks a zip file and prints out how many lines are in each compressed file (if you intend write code to walking through a zip file for real see "Walking through a zip file" in IO::Uncompress::Unzip ). Also, although this example uses the zlib-based comnpresion, the technique can be used by the other IO::Uncompress::* modules.
use strict;
use warnings;
use IO::File;
use IO::Uncompress::RawInflate qw(:all);
use constant ZIP_LOCAL_HDR_SIG => 0x04034b50;
use constant ZIP_LOCAL_HDR_LENGTH => 30;
my $file = $ARGV[0] ;
my $fh = new IO::File "<$file"
or die "Cannot open '$file': $!\n";
while (1)
{
my $sig;
my $buffer;
my $x ;
($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH
or die "Truncated file: $!\n";
my $signature = unpack ("V", substr($buffer, 0, 4));
last unless $signature == ZIP_LOCAL_HDR_SIG;
# Read Local Header
my $gpFlag = unpack ("v", substr($buffer, 6, 2));
my $compressedMethod = unpack ("v", substr($buffer, 8, 2));
my $compressedLength = unpack ("V", substr($buffer, 18, 4));
my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
my $filename_length = unpack ("v", substr($buffer, 26, 2));
my $extra_length = unpack ("v", substr($buffer, 28, 2));
my $filename ;
$fh->read($filename, $filename_length) == $filename_length
or die "Truncated file\n";
$fh->read($buffer, $extra_length) == $extra_length
or die "Truncated file\n";
if ($compressedMethod != 8 && $compressedMethod != 0)
{
warn "Skipping file '$filename' - not deflated $compressedMethod\n";
$fh->read($buffer, $compressedLength) == $compressedLength
or die "Truncated file\n";
next;
}
if ($compressedMethod == 0 && $gpFlag & 8 == 8)
{
die "Streamed Stored not supported for '$filename'\n";
}
next if $compressedLength == 0;
# Done reading the Local Header
my $inf = new IO::Uncompress::RawInflate $fh,
Transparent => 1,
InputLength => $compressedLength
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}The majority of the code above is concerned with reading the zip local header data. The code that I want to focus on is at the bottom.
while (1) {
# read local zip header data
# get $filename
# get $compressedLength
my $inf = new IO::Uncompress::RawInflate $fh,
Transparent => 1,
InputLength => $compressedLength
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}The call to IO::Uncompress::RawInflate creates a new filehandle $inf that can be used to read from the parent filehandle $fh, uncompressing it as it goes. The use of the InputLength option will guarantee that at most $compressedLength bytes of compressed data will be read from the $fh filehandle (The only exception is for an error case like a truncated file or a corrupt data stream).
This means that once RawInflate is finished $fh will be left at the byte directly after the compressed data stream.
Now consider what the code looks like without InputLength
while (1) {
# read local zip header data
# get $filename
# get $compressedLength
# read all the compressed data into $data
read($fh, $data, $compressedLength);
my $inf = new IO::Uncompress::RawInflate \$data,
Transparent => 1,
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}The difference here is the addition of the temporary variable $data. This is used to store a copy of the compressed data while it is being uncompressed.
If you know that $compressedLength isn't that big then using temporary storage won't be a problem. But if $compressedLength is very large or you are writing an application that other people will use, and so have no idea how big $compressedLength will be, it could be an issue.
Using InputLength avoids the use of temporary storage and means the application can cope with large compressed data streams.
One final point -- obviously InputLength can only be used whenever you know the length of the compressed data beforehand, like here with a zip file.
SEE ALSO
Compress::Zlib, IO::Compress::Gzip, IO::Uncompress::Gunzip, IO::Compress::Deflate, IO::Uncompress::Inflate, IO::Compress::RawDeflate, IO::Uncompress::RawInflate, IO::Compress::Bzip2, IO::Uncompress::Bunzip2, IO::Compress::Lzma, IO::Uncompress::UnLzma, IO::Compress::Xz, IO::Uncompress::UnXz, IO::Compress::Lzop, IO::Uncompress::UnLzop, IO::Compress::Lzf, IO::Uncompress::UnLzf, IO::Uncompress::AnyInflate, IO::Uncompress::AnyUncompress
File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib
AUTHOR
This module was written by Paul Marquess, pmqs@cpan.org.
MODIFICATION HISTORY
See the Changes file.
COPYRIGHT AND LICENSE
Copyright (c) 2005-2010 Paul Marquess. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.