Ive.qxd
7/9/04
11:04 AM
Page 252
The Material eXchange Format (MXF) and the Workflow Revolution By John G. S. Ive The Material eXchange Format (MXF) has captured the imagination of television engineers and is well on the way to becoming not only a SMPTE standard, but one of the most sought after new technology components in the drive towards ITbased production and distribution. The MXF file format has been developed as an interchange standard for video, audio, and metadata. In particular, metadata is key to changing workflows and improving operational efficiency. Many file formats exist today and are in regular use, raising the question “Why another file format?” This paper describes the unique MXF file format features resulting from the an original development with the specific needs of professional users in mind.
A
lthough taking longer than many imagined, the move toward IT-based production and distribution is progressing. In the current economic climate, cost-saving is one of the main motivators for change, in addition to the need for greater efficiency with more output from fewer resources and enhanced creativity. These priorities set a tough challenge, which cannot be achieved through the natural evolution of existing practices; therefore, a more fundamental change is required. To achieve the objectives at the system level, the correct technology building blocks need to be in place. Following successful work establishing the use of MPEG in the production environment, the Professional MPEG Forum turned its attention toward networking. A joint report of the SMPTE and (European Broadcasting Union (EBU) entitled “EBU/SMPTE Task Force for the Exchange of Programme Material as Bitstreams—Final Report, August 1998” identified several missing elements that needed to be addressed before fully functional networked environments could be introduced to satisfy the needs of the professional television industry. The most fundamental of these was a file format, supporting not only video and audio but also including extensive metadata capabilities to facilitate workflow changes fundamental to new operations. The accumulation and exploitation of metadata from acquisition to distribution is considered key to changing working practices, enabling smarter working. The workflow cycle is illustrated in Fig. 1. The first step taken by the Pro-MPEG Forum was to request inputs based on the technologies identified in the task force report, and then the second step was to identify and document user requirements. This was not easy because by the very revolutionary nature of the change, there was no best practice on which to base the work.
SMPTE
Journal, July/August 2004 •
Ive.qxd
7/9/04
11:04 AM
Page 253
THE MATERIAL EXCHANGE FORMAT (MXF) AND THE WORKFLOW REVOLUTION
Figure 1. The workflow cycle.
Nonetheless, a very useful set of requirements was established, a few of which are shown in Fig. 2.
What is the Material Exchange Format?
Metadata within MXF need not be static. Information about the video format, sampling, and so on will generally be fixed for the entire file. However, MXF can also contain metadata that has a specific relationship to the video and audio timeline. In this sense, virtual metadata “tracks” can be established in addition to video and audio tracks. Using this concept, timeline-based metadata can be synchronized with the relevant segments of audio and video. Examples could be dynamic camera pan, zoom, and focus information, or metadata related to scene changes. Returning to the video and audio components, otherwise known as “Essence,” MXF introduces several advanced features and capabilities in addition to complying with the fundamental user requirements. MXF acts as a wrapper and therefore does not in any way process the content itself. Because the creation of MXF files is based on a set of format rules, established as part of the proposed SMPTE standard, these rules can be applied not only to the wide range of current image formats, but are also applicable to future image coding strategies and resolutions. In summary, the MXF file format is: • primarily used for interchange and storage • a versatile and extensible wrapper • compression agnostic • operating system agnostic • metadata aware • streamable
Material eXchange Format (MXF) is an open file format for the interchange of audiovisual material with associated data and metadata. It has been designed and implemented with the aim of enabling intelligent filebased interoperability between content creation devices, workstations, and servers. By developing MXF from the beginning as a new file format, considerable thought has gone into the implementation and use of metadata, enabling powerful new tools for media management as well as improving content creation workflows. Currently, metadata is handled in a very “ad-hoc” way in the form of program planning notes, shooting information (often handwritten), tape labels, and information on disparate databases. This leads to metadata that becomes lost or inaccessible later in the production process and in many cases results in the need to re-create and re-enter the metadata, which is clearly inefficient and prone to errors. Metadata in MXF is divided into structural and descriptive parts. The structural part defines fundamental metadata needed to describe key parameters of the enclosed audio and video in the file and is compatible with the Advanced Authoring Format (AAF), thus allowing interoperability between the two. The descriptive part provides information for operational use and is, therefore, a major contributor to workflow improvements. Figure 2. Example of user requirements. SMPTE
Journal, July/August 2004 •
Ive.qxd
7/9/04
11:04 AM
Page 254
THE MATERIAL EXCHANGE FORMAT (MXF) AND THE WORKFLOW REVOLUTION
Figure 3. MXF documentation.
How is MXF Implemented?
number of bytes of the KLV packet, and the value is the content itself (Fig. 4). A file will contain content that a receiving device either does or does not need to understand. The key in each case determines this, and the length parameter tells the device how many bytes need to be ignored before the next key. Conversely, of course, the key also provides rapid detection of wanted subsets of information in a file. Operational Patterns have been created to respond to the various and, in some cases, divergent requirements of specific operational applications. The first operational pattern defined was “1A,” the most basic, in which one clip or program of continuous audio and video is wrapped together with its associated metadata. More complex operational patterns allow multiple clips to be wrapped and played out according to a defined sequence. The most complex levels allow for multiple clips with selective playout and sequencing using simple cut edits. To ensure maximum interoperability, it is highly likely that specific application areas will work primarily with one operational pattern or even mandate it. Figure 5 displays the operational patterns. The simplest (1A) is shown top left with increasing complexity and multiple clips making up the other patterns. The file package (FP) is the actual content of the file and the material package (MP) represents the way the file packages will be read out. Descriptive Metadata is probably one of the most discussed categories, which in turn is an area in which both end users and manufacturers have least experience. Traditionally, descriptive metadata might be no more than handwritten notes; a script, shooting annota-
The MXF format submitted to SMPTE consists of a suite of documents covering specific aspects of the format. This modular approach lends itself to the extensibility objectives. For example, as can be seen from the document layout in Fig. 3, there is a separate category for image coding (essence containers and mappings) with most popular implementations already defined. The most fundamental part is the format document itself, however, this is supported by documents including operational patterns, descriptive metadata plug-ins, and essence containers. For those requiring a more userfriendly approach to the format, the Engineering Guideline documents would be a good starting point. The format document describes the fundamental building blocks of the format and how they are assembled to provide specific solutions. As an example of this, MXF has followed SMPTE standards and implemented MXF using the “KLV” (key, length, value) structure. This has several advantages for media file operations. In summary, the content of an MXF file is coded throughout with KLV packets, which in turn are used to wrap individual elements attributed with a unique key. This applies to metadata and the video/audio content. The key identifies the Figure 4. MXF basic KLV structures. content, the length defines the SMPTE
Journal, July/August 2004 •
Ive.qxd
7/9/04
11:04 AM
Page 255
THE MATERIAL EXCHANGE FORMAT (MXF) AND THE WORKFLOW REVOLUTION uses the Unique Material Identifier (UMID) to provide for the linking and synchronization of remote metadata. When program files are traded or exchanged between organizations, then approved metadata would be re-inserted, based upon a standard template.
Generic Container
Figure 5. Operational patterns.
tions, good shot/bad shot information, or information held within disparate databases. With no coherent methodology to record and preserve metadata throughout the production and distribution chain, complex production and workflow efficiency improvements cannot be achieved. MXF therefore offers a set of tools to assist with workflow improvements and consistency. Using the SMPTE metadata dictionary as a starting point, MXF provides “DMS-1,” which is a general-purpose metadata model designed to satisfy many of the common aspects of production workflows. This metadata component is itself a “plug-in” with the option to implement other schemes compatible with the structural metadata. For specific applications, such as MPEG-7 and TV-Anytime that have fundamentally different structures, other mechanisms for carriage are under study. MXF is capable of carrying unlimited amounts of metadata, but operationally this may not be desirable. Within an organization, much of the metadata information will be created once and subsequently be applicable to multiple programs or program clips. In this case, it may be better to retain the information on a database and to link it to the relevant clips, thus requiring a single update in one location rather than a search through all files for all instances of the changed metadata. MXF SMPTE
Journal, July/August 2004 •
A further component of the MXF documentation is the “Generic Container,” which is the default essence container. As previously stated, MXF can adapt to any image coding strategy and image resolution. Already, documents have been implemented for uncompressed MPEG and DV in standard-definition and HDCAM for high-definition. Audio coding has also been addressed. Each of these image and audio formats uses the Generic Container with mapping to adapt, as new requirements are identified.
MXF in Action In the file and network domain, a fundamental operational change takes place compared with the characteristics of a tape environment. Because of the linear nature of tape, the physical recording on the media also dictates the way it is played out, usually in realtime. With this constraint, the concept of random tape access is not generally applicable as it is for file storage on diskbased media. In this case, the playout or reading of a file does not necessarily correspond to the physical stored sequence or the realtime speed of the material contained within the file. In MXF therefore, the physical storage structure and the playout of a virtual MXF file have to be considered separately. The playout will be derived from the physical contents, but under metadata control. Operational patterns and the metadata (already described) play a key role in defining the degrees of flexibility that exist with the file contents. However, several aspects of the MXF file’s physical structure are also important contributors to the flexibility.
Ive.qxd
7/9/04
11:04 AM
Page 256
THE MATERIAL EXCHANGE FORMAT (MXF) AND THE WORKFLOW REVOLUTION
Figure 6. MXF simple file contents.
Figure 6 shows a simplified example of the MXF physical structure. The lowest level is KLV coding, but this diagram represents the next level, highlighting some important functional components. More complex files can contain multiple essence containers, index tables, and metadata. Header and footer partitions establish the start and the end of the files. Header Metadata consists of both structural metadata (required for proper processing of the file) and descriptive metadata with human “targeted information” such as the production notes and other information previously described. Index Tables provide advanced operational benefits that may not be required in all cases. However, when working with very large files, reconstructing information relating to access points along the file will be slow and tedious. Therefore, where fast random access is required, index tables will be invaluable. The indexing approach efficiently accommodates fixed-length frames, in which there is a simple mathematical relationship between time and distance, and variable length (long GOP MPEG), where simple byte counts will not suffice. Essence Containers represent the location of the actual audio/video content as defined in the essence container documents.
professional production, storage, and distribution environment. In terms of workflow, extensive support for metadata will revolutionize operating processes and contribute to an increase in collaborative working through the sharing of media and associated metadata. The ability to carry metadata through the production and distribution chain together with the ability to interoperate between products and systems using the same framework will create a richer and more productive workflow environment. Broadcasters and professional end users are increasingly specifying MXF as a fundamental building block for new network installations based on its promise of interoperability and open standardization.
Bibliography Bruce Devlin and John Ive, “Working Together with MXF,” ProMPEG Publication, NAB, April 2003. Jim Wilkinson, Hans Hoffmann, Bruce Devlin, and Oliver Morgan, “MXF Pro-MPEG London Seminar, Jan. 2003. MXF, www.mxf.tv. Pro-MPEG Forum, www.pro-mpeg.org.
THE AUTHOR John Ive joined Sony Broadcast & Professional Europe in 1978 as general manager, advanced developments, leading an extensive team of research engineers. In 1985 he was promoted to director, product management, responsible for the planning and introduction of new broadcast and professional products. In 1990 he was made head of the company’s Broadcast & Professional Audio Business Division. He assumed his current position in 1998. An honors graduate in electronics, Ive first joined the BBC in London where he gained valuable operational experience. In 1974 he started a four-year period at the Independent Broadcasting Authority (IBA) working on digital video research. Throughout his career at Sony, Ive has contributed to the digitalization of the television industry. In recent years he has had similar involvement in the move towards IT-based production and high-definition television. Ive is a Fellow of SMPTE, a Fellow and Silver Medal holder of the Royal Television Society (RTS), a member of the FKTG in Germany, a member of the IEE, and a Chartered Engineer. He is also a director of the Professional MPEG Forum, an open industry organization working towards interoperability in networked environments.
Conclusion The Material Exchange Format set out to deliver a new generation of file format suitable for the converging worlds of professional AV and IT. The development has been an open industry initiative led by the Professional MPEG Forum in association with organizations such as SMPTE, the EBU, and the AAF Association. MXF has outstanding industry support from both manufacturers and end users. The first phase of the documentation is robust and stable with SMPTE standardization complete. Practical implementations have been announced and are currently available. Uniquely, MXF incorporates a wide range of userdemanded features developed specifically for the
Presented at the SMPTE 2003 Conference of the Australia Section, June 30July 4, 2003. Copyright © 2004 by SMPTE.
SMPTE
Journal, July/August 2004 •
