File format
A file format is the way that information is encoded for storage in a computer file. It may describe the encoding at various levels of abstraction including low-level bit and byte layout as well high-level organization such as markup and tabular structure. A file format may be standardized or it can be an ad hoc convention.
Some file formats are designed for very particular types of data: PNG files, for example, store bitmapped images using lossless data compression. Other file formats, however, are designed for storage of several different types of data: the Ogg format can act as a container for different types of multimedia including any combination of audio and video, with or without text, and metadata. A text file can contain any stream of characters, including possible control characters, and is encoded in one of various character encoding schemes. Some file formats, such as HTML, scalable vector graphics, and the source code of computer software are text files with defined syntaxes that allow them to be used for specific purposes.
Specification
Some file formats have a published specification describing the format and possibly how to verify the correctness of data in that format. Such a document is not available for every format. Sometimes, a format is defined de facto by the behavior of the program that accesses the file.If there is no specification available, a developer might reverse engineer the format by inspecting files in that format or acquire the specification for a fee and by signing a non-disclosure agreement. Due to the time and money cost of these approaches, file formats with publicly available specifications tend to be supported by more programs.
Intellectual property protection
law can be used to protect the intellectual property inherent in a file format. Although a patent for a file format is not directly permitted under US law, some formats encode data using a patented algorithm. For example, prior to 2004, using compression with the GIF file format required the use of a patented algorithm, and though the patent owner did not initially enforce their patent, they later began collecting royalty fees. This has resulted in a significant decrease in the use of GIFs, and is partly responsible for the development of the alternative PNG format. However, the GIF patent expired in the US in mid-2003, and worldwide in mid-2004.Identification
Both users and applications need to identify a file's format so that the file can be used appropriately. Generally, the methods for identification vary by operating system, with each approach having its advantages and disadvantages.Filename extension
One popular method used by many operating systems, including Windows, macOS, CP/M, MS-DOS, VMS, and VM/CMS, is to indicate the format of a file with a suffix of the file name, known as the extension. For example, an HTML document is identified by a file name that ends with or, and a GIF image by.In the now-antiquated FAT file system, file names were limited to eight characters for the base name plus a three-character extension, known as an 8.3 filename. Due to the prevalence of this naming scheme, many formats still use three-character extensions even though modern systems support longer extensions. Since there is no standardized list of extensions, more than one format can use the same extension especially for three-letter extensions since there is a limited number of three-letter combinations. This situation can confuse both users and applications.
One implication of indicating the file type with the extension is that the users and applications can be tricked into treating a file as a different format simply by renaming it. For example, an HTML file can be treated as plain text by adding extension. Although this strategy is useful, it can be confusing to less technical users who accidentally make a file unusable. To try to avoid this scenario, Windows and macOS support hiding the extension.
Hiding the extension, however, can create the appearance of multiple files with the same name in the same folder, which is confusing for people. For example, an image may be needed both in Encapsulated PostScript| format and.png format and one might name them with the same base name. With extensions hidden they appear to have the same name:.
Hiding extensions can also pose a security risk. For example, a malicious user could create an executable program with an innocent name such as "". The "" would be hidden and an unsuspecting user would see "", which would appear to be a JPEG image, usually unable to harm the machine. However, the operating system would still see the "" extension and run the program, which would then be able to cause harm to the computer. The same is true with files with only one extension: as it is not shown to the user, no information about the file can be deduced without explicitly investigating the file. To further trick users, it is possible to store an icon inside the program, in which case some operating systems' icon assignment for the executable file would be overridden with an icon commonly used to represent JPEG images, making the program look like an image. Extensions can also be spoofed: some Microsoft Word macro viruses create a Word file in template format and save it with a extension. Since Word generally ignores extensions and looks at the format of the file, these would open as templates, execute, and spread the virus. This represents a practical problem for Windows systems where extension-hiding is turned on by default.
Internal metadata
A file's format may be indicated inside the file itself either as information intended for this purpose or as identifiable data within the format that can be used for identification even though that is not its intended purpose.Often intentionally placed information is located at the beginning of a file since this is relatively easy to read from a file both by users and applications. When the information at the beginning of the file is a structure that contains other metadata, then the structure is often called a file header. When the file starts with a relatively small datum that only indicates the format, then it is often called a magic number.
File header
The metadata contained in a file header are usually stored at the start of the file, but might be present in other areas too, often including the end, depending on the file format or the type of data contained. Character-based files usually have character-based headers, whereas binary formats usually have binary headers, although this is not a rule. Text-based file headers usually take up more space, but being human-readable, they can easily be examined by using simple software such as a text editor or a hexadecimal editor.As well as indicating the file format, file headers may contain metadata about the file and its contents. For example, most image files store information about image format, size, resolution and color space, and optionally authoring information such as who made the image, when and where it was made, what camera model and photographic settings were used, and so on. Such metadata may be used by software reading or interpreting the file during the loading process and afterwards.
File headers may be used by an operating system to quickly gather information about a file without loading it all into memory, but doing so uses more of a computer's resources than reading directly from the directory information. For instance, when a graphic file manager has to display the contents of a folder, it must read the headers of many files before it can display the appropriate icons, but these will be located in different places on the storage medium thus taking longer to access. A folder containing many files with complex metadata such as thumbnail information may require considerable time before it can be displayed.
If a header is binary hard-coded such that the header itself needs complex interpretation in order to be recognized, especially for metadata content protection's sake, there is a risk that the file format can be misinterpreted. It may even have been badly written at the source. This can result in corrupt metadata which, in extremely bad cases, might even render the file unreadable.
A more complex example of file headers are those used for wrapper file formats.
Magic number
One way to incorporate file type metadata is to store a "magic number" inside the file itself. Originally, this term was used for 2-byte identifiers at the start of files, but since any binary sequence can be regarded as a number, any feature of a file format which uniquely distinguishes it can be used for identification. GIF images, for instance, always begin with the ASCII representation of eitherGIF87a or GIF89a, depending upon the standard to which they adhere. Many file types, especially plain-text files, are harder to spot by this method. HTML files, for example, might begin with the string <html>, or an appropriate document type definition that starts with <!DOCTYPE html, or, for XHTML, the XML identifier, which begins with <?xml. The files can also begin with HTML comments, random text, or several empty lines, but still be usable HTML.The magic number approach offers better guarantees that the format will be identified correctly, and can often determine more precise information about the file. Since reasonably reliable "magic number" tests can be fairly complex, and each file must effectively be tested against every possibility in the magic database, this approach is relatively inefficient, especially for displaying large lists of files. Also, data must be read from the file itself, increasing latency as opposed to metadata stored in the directory. Where file types do not lend themselves to recognition in this way, the system must fall back to metadata. It is, however, the best way for a program to check if the file it has been told to process is of the correct format: while the file's name or metadata may be altered independently of its content, failing a well-designed magic number test is a pretty sure sign that the file is either corrupt or of the wrong type. On the other hand, a valid magic number does not guarantee that the file is not corrupt or is of a correct type.
So-called shebang lines in script files are a special case of magic numbers. There, the magic number consists of human-readable text within the file that identifies a specific interpreter and options to be passed to it.
Another operating system using magic numbers is AmigaOS, where magic numbers were called "Magic Cookies" and were adopted as a standard system to recognize executables in Hunk executable file format and also to let single programs, tools and utilities deal automatically with their saved data files, or any other kind of file types when saving and loading data. This system was then enhanced with the Amiga standard Datatype recognition system. Another method was the FourCC method, originating in OSType on Macintosh, later adapted by Interchange File Format and derivatives.