--- /dev/null
+@c This is part of the paxutils manual.
+@c Copyright (C) 2006 Free Software Foundation, Inc.
+@c This file is distributed under GFDL 1.1 or any later version
+@c published by the Free Software Foundation.
+
+The notion of sparse file, and the ways of handling it from the point
+of view of @GNUTAR{} user have been described in detail in
+@ref{sparse}. This chapter describes the internal format @GNUTAR{}
+uses to store such files.
+
+The support for sparse files in @GNUTAR{} has a long history. The
+earliest version featuring this support that I was able to find was 1.09,
+released in November, 1990. The format introduced back then is called
+@dfn{old GNU} sparse format and in spite of the fact that its design
+contained many flaws, it was the only format @GNUTAR{} supported
+until version 1.14 (May, 2004), which introduced initial support for
+sparse archives in @acronym{PAX} archives (@pxref{posix}). This
+format was not free from design flows, either and it was subsequently
+improved in versions 1.15.2 (November, 2005) and 1.15.92 (June,
+2006).
+
+In addition to GNU sparse format, @GNUTAR{} is able to read and
+extract sparse files archived by @command{star}.
+
+The following subsections describe each format in detail.
+
+@menu
+* Old GNU Format::
+* PAX 0:: PAX Format, Versions 0.0 and 0.1
+* PAX 1:: PAX Format, Version 1.0
+@end menu
+
+@node Old GNU Format
+@appendixsubsec Old GNU Format
+
+The format introduced some time around 1990 (v. 1.09). It was
+designed on top of standard @code{ustar} headers in such an
+unfortunate way that some of its fields overwrote fields required by
+POSIX.
+
+An old GNU sparse header is designated by type @samp{S}
+(@code{GNUTYPE_SPARSE}) and has the following layout:
+
+@multitable @columnfractions 0.10 0.10 0.20 0.20 0.40
+@headitem Offset @tab Size @tab Name @tab Data type @tab Contents
+@item 0 @tab 345 @tab @tab N/A @tab Not used.
+@item 345 @tab 12 @tab atime @tab Number @tab @code{atime} of the file.
+@item 357 @tab 12 @tab ctime @tab Number @tab @code{ctime} of the file .
+@item 369 @tab 12 @tab offset @tab Number @tab For
+multivolume archives: the offset of the start of this volume.
+@item 381 @tab 4 @tab @tab N/A @tab Not used.
+@item 385 @tab 1 @tab @tab N/A @tab Not used.
+@item 386 @tab 96 @tab sp @tab @code{sparse_header} @tab (4 entries) File map.
+@item 482 @tab 1 @tab isextended @tab Bool @tab @code{1} if an
+extension sparse header follows, @code{0} otherwise.
+@item 483 @tab 12 @tab realsize @tab Number @tab Real size of the file.
+@end multitable
+
+Each of @code{sparse_header} object at offset 386 describes a single
+data chunk. It has the following structure:
+
+@multitable @columnfractions 0.10 0.10 0.20 0.60
+@headitem Offset @tab Size @tab Data type @tab Contents
+@item 0 @tab 12 @tab Number @tab Offset of the
+beginning of the chunk.
+@item 12 @tab 12 @tab Number @tab Size of the chunk.
+@end multitable
+
+If the member contains more than four chunks, the @code{isextended}
+field of the header has the value @code{1} and the main header is
+followed by one or more @dfn{extension headers}. Each such header has
+the following structure:
+
+@multitable @columnfractions 0.10 0.10 0.20 0.20 0.40
+@headitem Offset @tab Size @tab Name @tab Data type @tab Contents
+@item 0 @tab 21 @tab sp @tab @code{sparse_header} @tab
+(21 entires) File map.
+@item 504 @tab 1 @tab isextended @tab Bool @tab @code{1} if an
+extension sparse header follows, or @code{0} otherwise.
+@end multitable
+
+A header with @code{isextended=0} ends the map.
+
+@node PAX 0
+@appendixsubsec PAX Format, Versions 0.0 and 0.1
+@UNREVISED{}
+
+There are two formats available in this branch. The version @code{0.0}
+is the initial version of sparse format used by @command{tar}
+versions 1.14--1.15.1. The sparse file map is kept in extended
+(@code{x}) PAX header variables:
+
+@table @code
+@item GNU.sparse.size
+Real size of the stored file
+
+@item GNU.sparse.numblocks
+Number of blocks in the sparse map
+
+@item GNU.sparse.offset
+Offset of the data block
+
+@item GNU.sparse.numbytes
+Size of the data block
+@end table
+
+The latter two variables repeat for each data block, so the overall
+structure is like this:
+
+@smallexample
+@group
+GNU.sparse.size=@var{size}
+GNU.sparse.numblocks=@var{numblocks}
+repeat @var{numblocks} times
+ GNU.sparse.offset=@var{offset}
+ GNU.sparse.numbytes=@var{numbytes}
+end repeat
+@end group
+@end smallexample
+
+This format presented the following two problems:
+
+@enumerate 1
+@item
+Whereas the POSIX specification allows a variable to appear multiple
+times in a header, it requires that only the last occurrence be
+meaningful. Thus, multiple ocurrences of @code{GNU.sparse.offset} and
+@code{GNU.sparse.numbytes} are conficting with the POSIX specs.
+
+@item
+Attempting to extract such archives using a third-party @command{tar}s
+results in extraction of sparse files in @emph{compressed form}. If
+the @command{tar} implementation in question does not support POSIX
+format, it will also extract a file containing extension header
+attributes. This file can be used to expand the file to its original
+state. However, posix-aware @command{tar}s will usually ignore the
+unknown variables, which makes restoring the file much more
+difficult@FIXME-xref{how to extract sparse file using third-party @command{tar}s}.
+@end enumerate
+
+@GNUTAR{} 1.15.2 introduced sparse format version @code{0.1}, which
+attempted to solve these problems. As its predecessor, this format
+stores sparse map in the extended POSIX header. It retains
+@code{GNU.sparse.size} and @code{GNU.sparse.numblocks} variables, but
+instead of @code{GNU.sparse.offset}/@code{GNU.sparse.numbytes} pairs
+it uses a single variable:
+
+@table @code
+@item GNU.sparse.map
+Map of non-null data chunks. It is a string consisting of
+comma-separated values "@var{offset},@var{size}[,@var{offset-1},@var{size-1}...]"
+@end table
+
+To address the 2nd problem, the @code{name} field in @code{ustar}
+is replaced with a special name, constructed using the following pattern:
+
+@smallexample
+%d/GNUSparseFile.%p/%f
+@end smallexample
+
+The real name of the sparse file is stored in the variable
+@code{GNU.sparse.name}. Thus, those @command{tar} implementations
+that are not aware of GNU extensions will at least extract the files
+into separate directories, giving the user a possibility to expand it
+afterwards @FIXME-ref{how to extract sparse file using third-party
+@command{tar}s}.
+
+The resulting @code{GNU.sparse.map} string can be @emph{very} long.
+Although POSIX does not impose any limit on the length of a @code{x}
+header variable, this possibly can confuse some tars.
+
+@node PAX 1
+@appendixsubsec PAX Format, Version 1.0
+@UNREVISED{}
+
+The version @code{1.0} of sparse format was introduced with @GNUTAR{}
+1.15.92. Its main objective was to make the resulting file
+extractable with little effort even by non-posix aware @command{tar}
+implementations. Starting from this version, the extended header
+preceding a sparse member always contains the following variables that
+identify the format being used:
+
+@table @code
+@item GNU.sparse.major
+Major version
+
+@item GNU.sparse.minor
+Minor version
+@end table
+
+The @code{name} field in @code{ustar} header contains a special name,
+constructed using the following pattern:
+
+@smallexample
+%d/GNUSparseFile.%p/%f
+@end smallexample
+
+The real name of the sparse file is stored in the variable
+@code{GNU.sparse.name}. The real size of the file is stored in the
+variable @code{GNU.sparse.realsize}.
+
+The sparse map itself is stored in the file data block, preceding the actual
+file data. It consists of a series of octal numbers of arbitrary length, delimited
+by newlines. The map is padded with nulls to the nearest block boundary.
+
+The first number gives the number of entries in the map. Following are map entries,
+each one consisting of two numbers giving the offset and size of the
+data block it describes.
+
+The format is designed in such a way that non-posix aware tars and tars not
+supporting @code{GNU.sparse.*} keywords will extract each sparse file
+in its condensed form with the file map prepended and will place it
+into a separate directory. Then, using a simple program it would be
+possible to expand the file to its original form even without GNU tar.
+@FIXME-xref{how to extract sparse file using third-party
+@command{tar}s}. @FIXME{Write the program and give its URL here}.
+
projects / chaz / tar / commitdiff
