Tar (computing)
In computing, tar is a shell command for combining multiple computer files into a single archive file. It was originally developed for magnetic tape computer storage reading and writing data for a sequential I/O device with no file system, and the name is short for the format description "tape archive". When stored in a file system, a file that tar reads and writes is often called a tarball.
A tarball contains metadata for the contained files including the name, ownership, timestamps, permissions and directory organization. As a file containing other files with associated metadata, a tarball is useful for software distribution and backup.
POSIX abandoned tar in favor of [|pax], yet tar continues to have widespread use.
History
The command was introduced to Unix in January 1979, replacing the tp program. The file structure was standardized in POSIX.1-1988 and later POSIX.1-2001, and became a format supported by most modern file archiving utilities. The tar command was abandoned in POSIX.1-2001 in favor of pax, which was to support the ustar file format, and tar was indicated for withdrawal in favor of pax at least since 1994. None-the-less, many operating systems today include tools for tar files, as well as tools to compress and decompress them, such as xz, gzip, and bzip2.The command was ported to the IBM i operating system.
BSD-tar has been in Windows since 2018, and there are other third-party tools available for Windows.
Rationale
Many historic tape drives read and write variable-length data blocks, leaving significant wasted space on the tape between blocks. Some tape drives support only fixed-length data blocks. Also, when writing to any medium such as a file system or network, it takes less time to write one large block than many small blocks. Therefore, the tar command writes data in records of many 512 B blocks. The user can specify a blocking factor, which is the number of blocks per record. The default is 20, producing 10 KiB records.File format
There are multiple tar file formats, including historical and current ones. Two tar formats are codified in POSIX: ustar and pax. Not codified but still in current use is the GNU tar format.A tar archive consists of a series of file objects, hence the popular term tarball, referencing how a tarball collects objects of all kinds that stick to its surface. Each file object includes any file data, and is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes. The original tar implementation did not care about the contents of the padding bytes, and left the buffer data unaltered, but most modern tar implementations fill the extra space with zeros. The end of an archive is marked by at least two consecutive zero-filled records. The final block of an archive is padded out to full length with zeros.
Header
The file header record contains metadata about a file. To ensure portability across different architectures with different byte orderings, the information in the header record is encoded in ASCII. Thus if all the files in an archive are ASCII text files, and have ASCII names, then the archive is essentially an ASCII text file.The fields defined by the original Unix tar format are listed in the table below. The link indicator/file type table includes some modern extensions. When a field is unused it is filled with NUL bytes. The header uses 257 bytes, then is padded with NUL bytes to make it fill a 512 byte record. There is no "magic number" in the header, for file identification.
Pre-POSIX.1-1988 tar header:
| Field offset | Field size | Field |
| 0 | 100 | File path and name |
| 100 | 8 | File mode |
| 108 | 8 | Owner's numeric user ID |
| 116 | 8 | Group's numeric user ID |
| 124 | 12 | File size in bytes |
| 136 | 12 | Last modification time in numeric Unix time format |
| 148 | 8 | Checksum for header record |
| 156 | 1 | Link indicator |
| 157 | 100 | Name of linked file |
The pre-POSIX.1-1988 Link indicator field can have the following values:
| Value | Meaning |
| '0' or | Normal file |
| '1' | Hard link |
| '2' | Symbolic link |
Some pre-POSIX.1-1988 tar implementations indicated a directory by having a trailing slash in the name.
Numeric values are encoded as octal numbers using ASCII digits, with leading zeroes. Whilst this choice may seem counterintuitive in modern times when the four bit hexadecimal notation is generally preferred because register sizes and address widths in computers are almost always powers of two and bytes have standardized to be octets, Unix was originally developed for the PDP-7, which uses an 18-bit CPU and a six-bit character code, making the three bit octal notation more desirable.
For historical reasons, a final NUL or space character should also be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, in 2001 star introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.
The checksum is calculated by taking the sum of the unsigned byte values of the header record with the eight checksum bytes taken to be ASCII spaces. It is stored as a six digit octal number with leading zeroes followed by a NUL and then a space. Various implementations do not adhere to this format. In addition, some historic tar implementations treated bytes as signed. Implementations typically calculate the checksum both ways, and treat it as good if either the signed or unsigned sum matches the included checksum.
Unix filesystems support multiple links for the same file. If several such files appear in a tar archive, only the first one is archived as a normal file; the rest are archived as hard links, with the "name of linked file" field set to the first one's name. On extraction, such hard links should be recreated in the file system.
UStar format
Most modern tar programs read and write archives in the UStar format, introduced by the POSIX IEEE P1003.1 standard from 1988. It introduced additional header fields. Older tar programs will ignore the extra information, while newer programs will test for the presence of the "ustar" string to determine if the new format is in use.The UStar format allows for longer file names and stores additional information about each file. The maximum filename size is 255, but it is split among a preceding path "filename prefix" and the filename itself, so can be much less.
| Field offset | Field size | Field |
| 0 | 156 | ' |
| 156 | 1 | Type flag |
| 157 | 100 | ' |
| 257 | 6 | UStar indicator, "ustar", then NUL |
| 263 | 2 | UStar version, "00" |
| 265 | 32 | Owner user name |
| 297 | 32 | Owner group name |
| 329 | 8 | Device major number |
| 337 | 8 | Device minor number |
| 345 | 155 | Filename prefix |
The type flag field can have the following values:
| Value | Meaning |
| '0' or | Normal file |
| '1' | Hard link |
| '2' | Symbolic link |
| '3' | Character special |
| '4' | Block special |
| '5' | Directory |
| '6' | FIFO |
| '7' | Contiguous file |
| 'g' | Global extended header with meta data |
| 'x' | Extended header with metadata for the next file in the archive |
| 'A'–'Z' | Vendor specific extensions |
| All other values | Reserved for future standardization |
POSIX.1-1988 vendor specific extensions using link flag values 'A'–'Z' partially have a different meaning with different vendors and thus are seen as outdated and replaced by the POSIX.1-2001 extensions that also include a vendor tag.
Type '7' is formally marked as reserved in the POSIX standard, but was meant to indicate files which ought to be contiguously allocated on disk. Few operating systems support creating such files explicitly, and hence most tar programs do not support them, and will treat type 7 files as if they were type 0. An exception is older versions of GNU tar, when running on the MASSCOMP RTU operating system, which supported an O_CTG flag to the open function to request a contiguous file; however, that support was removed from GNU tar version 1.24 onwards.
POSIX.1-2001/pax
In 1997, Sun proposed a method for adding extensions to the tar format. This method was later accepted for the POSIX.1-2001 standard. This format is known as extended tar format or pax format. The new tar format allows users to add any type of vendor-tagged vendor-specific enhancements. The following tags are defined by the POSIX standard:- atime, mtime: all timestamps of a file in arbitrary resolution
- path: path names of unlimited length and character set coding
- linkpath: symlink target names of unlimited length and character set coding
- uname, gname: user and group names of unlimited length and character set coding
- size: files with unlimited size
- uid, gid: userid and groupid without size limitation
- a character set definition for path names and user/group names
The pax format is designed so that all implementations able to read the UStar format will be able to read the pax format as well. The only exceptions are files that make use of extended features, such as longer file names. For compatibility, these are encoded in the tar files as special or type files, typically under a directory. A pax-supporting implementation would make use of the information, while non-supporting ones like 7-Zip would process them as additional files.