Scalable Real-Time Computing in the Solaris Operating Environment A Technical White Paper

Sun Microsystems, Inc. 901 San Antonio Road Palo Alto, CA 94303 1 (800) 786.7638 1.512.434.1511

™

Copyright 2000 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303 U.S.A. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, CacheFS, Java, Java Studio, Java WorkShop, NFS, Solaris, Solaris Resource Manager, SunOS, SunATM, Sun WorkShop, Sun Visual WorkShop, Trusted Solaris, and Ultra are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements. RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015(b)(6/95) and DFAR 227.7202-3(a). DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. Copyright 2000 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, Californie 94303 Etats-Unis. Tous droits réservés. Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et la décompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a. Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun. Des parties de ce produit pourront être dérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd. Sun, Sun Microsystems, le logo Sun, CacheFS, Java, Java Studio, Java WorkShop, NFS, Solaris, Solaris Resource Manager, SunOS, SunATM, Sun WorkShop, Sun Visual WorkShop, Trusted Solaris, et Ultra sont des marques de fabrique ou des marques déposées, ou marques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays. Toutes les marques SPARC sont utilisées sous licence et sont des marques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d’autres pays. Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc. L’interface d’utilisation graphique OPEN LOOK et Sun™ a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés. Sun reconnaît les efforts de pionniers de Xerox pour la recherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industrie de l’informatique. Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre se conforment aux licences écrites de Sun. CETTE PUBLICATION EST FOURNIE “EN L’ETAT” ET AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, N’EST ACCORDEE, Y COMPRIS DES GARANTIES CONCERNANT LA VALEUR MARCHANDE, L’APTITUDE DE LA PUBLICATION A REPONDRE A UNE UTILISATION PARTICULIERE, OU LE FAIT QU’ELLE NE SOIT PAS CONTREFAISANTE DE PRODUIT DE TIERS. CE DENI DE GARANTIE NE S’APPLIQUERAIT PAS, DANS LA MESURE OU IL SERAIT TENU JURIDIQUEMENT NUL ET NON AVENU.

Please Recycle

Contents A New Look at Real-Time Computing ......................................................1 The Growing Demand for Real-Time Applications ...............................2 Scalable Real-Time Capabilities in the Solaris Operating Environment ...........................................................................4 A Bounded Interval Response Time ...................................................5 A Mature Foundation for Real-Time Computing ...............................6 Real-Time Innovations in the Solaris Operating Environment ..................8 Multithreaded Kernel ..............................................................................8 A Scalable, Flexible Model ................................................................8 Processor Binding and Processor Sets ................................................10 Processor Sets .................................................................................10 Interrupt Sheltering .........................................................................10 Priority-Based Scheduling ......................................................................11 Global Priority Model .........................................................................11 The Real-Time Scheduling Class .......................................................12 Priority Inversion ............................................................................13 Interrupts .........................................................................................13 Precision Clocks and Timers ..................................................................14 Standard Clocks and Timers ...............................................................14 Configurable Hertz .............................................................................14 High-Resolution Hardware-based Timers ..........................................15 CLOCK_HIGHRES on UltraSPARC .............................................15 CLOCK_HIGHRES on Other Platforms ........................................15

Timestamps .........................................................................................16 Network Time Protocol (NTP) ...........................................................16 Features for Improved Determinism .......................................................16 Memory Locking ................................................................................16 Early Binding ......................................................................................17 Priority STREAMS .............................................................................17 A Compelling Development and Run-Time Environment .........................18 Networking and I/O Technology for Real-Time Applications ...............18 64-Bit Large Files ...............................................................................18 Raw and UFS Direct I/O .....................................................................19 Kernel Sockets ....................................................................................19 TCP Large Windows ..........................................................................19 TCP Selective Acknowledgment ........................................................20 Zero Copy TCP/Hardware Checksum ................................................20 Source Compatibility ..............................................................................20 Powerful Development Tools .................................................................21 Debugging — Trace Neutral Format (TNF) .......................................21 Sun WorkShop Products .....................................................................22 Java Applications Development .........................................................23 Real-Time Java Implementation .....................................................23 Summary: Sun’s Real-Time Vision ............................................................24 Appendix ....................................................................................................25 POSIX Real-time Extensions ...............................................................25 Supported POSIX.1b Extensions ........................................................26 Supported POSIX.1c Extensions ........................................................27

CHAPTER

1

A New Look at Real-Time Computing As we seek to make faster and better-informed decisions, real-time applications will play an increasingly critical role. Military, scientific, and civil organizations are already deploying a proliferation of sensors — from devices that measure temperature, vibration, pressure, air quality, and radioactivity to those that deliver audio and video signals. This planetary veil of sensors requires a wealth of real-time applications — from monitoring hydrologic flow and air pollution to detecting small-scale nuclear tests and fallout, improving air safety, and predicting seismic and weather events. Commercial enterprises are also finding new applications for real-time technology. The telecommunications, financial, and automotive industries as well as others are demanding predictable application behavior and higher levels of quality of service (QoS). Together, these significant new demands stretch the applicability of traditional real-time approaches. In the past, real-time application development has been problematic and expensive because it required development efforts targeted at specialized — and often dedicated — uniprocessor delivery platforms. These non-standard environments, with their limited development tools, often resulted in non-portable, special-purpose application code that was difficult to write and costly to maintain. Worse, because delivery platforms also tended to be special-purpose, applications that outgrew their host often had to be completely re-written in order to support a more powerful platform.

1

™

In contrast, the Solaris 8 Operating Environment offers a mature, scalable, real-time environment where applications have access to dedicated resources on powerful multiprocessor systems. Full POSIX compliance delivered on top of a preemptive, multithreaded kernel provides a rich complement of real-time capabilities, and enables developers to write portable real-time code. The Solaris 8 Operating Environment also provides a wealth of features that are key to real-time programmers, including support for applications with real-time and time-share components (mixed-mode applications), selectable security (C2 and B1+), and 32-/64-bit application support. Full source code compatibility within the Solaris Operating Environment enables real-time application programmers to access all of these features from a single source tree for deployment on powerful UltraSPARC and Intel uniprocessor or multiprocessor platforms. ™

This document has been prepared for release with the Solaris 8 Operating Environment. Chapter 1 provides an introduction to real-time concepts, markets, and real-time capabilities, while Chapter 2 examines the real-time architectural foundation of the Solaris Operating Environment. Chapter 3 lists additional features useful to developers of real-time applications. The results of real-time performance testing will be provided in a subsequent edition of this paper.

The Growing Demand for Real-Time Applications Far from a niche marketplace, real-time applications today play an important role in many different markets.

Military Modern military advantage depends on rapid, secure, real-time information superiority — gathering data from a wide variety of real-time sensors (satellite, air, surface, subsurface) and delivering it to analysis capabilities and weapons systems. With rapid deployment, dynamic configuration of weapons systems, and greater security of primary concern, the military will be required to move to scalable and secure, distributed real-time operational systems (Vision 2010). These futuristic systems will dynamically integrate sensors, C2 security, and shooters into large, virtual, network-centric systems that cross service boundaries. The Solaris Operating Environment provides the foundation technologies to make this vision of true network-centric defense a reality.

2

Scalable Real-Time Computing in the Solaris Operating Environment

Telecommunications Modern consumers are demanding real-time access to information from a vast array of devices, including computers, set-top boxes, PDAs, pagers, and wireless phones. As media consolidation continues, telecommunications and cable companies will have to deploy greater bandwidth and real-time delivery of services in both their land-line and wireless networks. These networks — and the applications supporting them — will need to scale to deliver terabytes of time-synchronous information for compelling applications such as video on demand.

Aviation Air travel continues to increase, resulting in greater congestion and contention for limited routing space. Larger planes can only go so far in serving greater numbers of passengers and cargo; air traffic control systems are being modernized to support higher traffic loads and more efficient routing decisions. Improved real-time capabilities will increase safety and efficiency by identifying dangerous situations and enabling flexible routing to relieve congestion and prevent collisions. Real-time computing will play a critical role in free flight, the next wave in air traffic control intended to ease congested flight paths.

Automotive In the quest for anywhere, anytime information, the automotive industry’s goal will be to act as the ISP for nomadic computing by providing a wealth of wireless services. Commuters will receive real-time data, GPS location information, and operational data via ground stations and satellites. Drivers will indicate an end destination while ground servers accepting GPS information relay real-time updates to video screens in the vehicle, routing the driver around accidents, road work, or traffic congestion in real time.

Financial As business pressures demand quicker decisions, the guarantee of timely information becomes essential to success. In the enterprise in general, the need to provide bounded transactions and higher levels of QoS are the next logical step. For instance, a CIO will want to know exactly how many transactions the IT architecture can support in order to plan a backlog and provide timely shipment. Electronic commerce and just-in-time manufacturing will make the entire enterprise more finegrained and predictable. Having the information and ability to expand network bandwidth or CPU horsepower on demand will become a powerful business tool.

A New Look at Real-Time Computing

3

Scientific Real-time systems are already heavily used in scientific communities for monitoring and instrument control. The growing numbers of deployed sensors and new distributed, real-time techniques continue to suggest new solutions to old problems. For example, seismic information is increasingly distributed on a vast scale, enabling scientists to verify even small-scale nuclear detonations, as well as helping smaller remote countries prepare for weather or seismic disasters.

Scalable Real-Time Capabilities in the Solaris Operating Environment Sun has long appreciated the importance of real-time applications. The Solaris Operating Environment was architected with a solid real-time foundation from its inception in 1987. In particular, Sun realized the power of combining both real-time and time-share capabilities in a volume, commercial off-the-shelf (COTS) operating system. This approach gives real-time systems access to extensive application availability, a flexible development environment for native real-time applications, and standards-based application portability. Beyond these advantages, the Solaris Operating Environment allows real-time deployment systems to scale to meet increasing application demands. As computational load increases, additional processors can be added to the system for either time-share or real-time tasks — providing not only time-share, but real-time enterprise scalability. Systems can even be dynamically reconfigured to provide proportionally more dedicated, real-time computation resources on an as-needed basis. No matter how attractive these advantages are, real-time applications have stringent timing requirements that must be taken into consideration in order for an operating system to be considered for deployment.

4

Scalable Real-Time Computing in the Solaris Operating Environment

A Bounded Interval Response Time To be useful for real-time applications, an operating system must provide a bounded interval response time — defined as the time elapsed from when a high-level interrupt is received until the system context is switched and the first instruction of the new application runs. Real-time application programmers need the interval response time to be bounded so that they are effectively guaranteed a maximum time in which an interrupt will be serviced.

High Level Interrupt

Processor Receives Interrupt

OS Response

Dispatch

User Process Execution Begins Time

Interrupt Latency

Interrupt Service

Interrupt Response (Driver)

Preemption Latency (kernel/user)

Context Switch

Dispatch Latency (Kernel)

Interval Response Time

FIGURE 1

Interval response components in the Solaris Operating Environment

Figure 1 illustrates the components that comprise the interval response time. Interrupt latency is the time required for the interrupt to propagate through the hardware from its source to the processor’s interrupt pin. The interrupt service interval spans the time from when the processor receives the interrupt and the instant when the operating system begins responding to the interrupt. Together, the interrupt latency and interrupt service interval make up the driver’s interrupt response. Once the operating system has serviced the interrupt, the kernel’s preemption latency is the time needed for the scheduler to determine which thread should run and the time for the new thread to be dispatched. The kernel then saves the state of the interrupted thread or process, loads the context for the preempting thread (context switch), and begins execution. Configuration details and performance information for Sun’s interval response time will be provided in a future version of this document.

A New Look at Real-Time Computing

5

A Mature Foundation for Real-Time Computing Achieving a bounded response time requires a robust and mature real-time foundation. The Solaris Operating Environment offers significant advantages for developing and deploying real-time applications.

Standard POSIX Interface Compliance Beyond merely including POSIX library bindings (as some operating systems do), the Solaris 8 Operating Environment implements a full POSIX 1003.1b (formerly 1003.4) real-time operating environment with support for POSIX 1003.1c (formerly 1003.4a) threaded applications. Details on the POSIX extensions supported by the Solaris Operating Environment can be found in Appendix A.

Fully Preemptable, Multithreaded Kernel The Solaris 8 Operating Environment features a fully symmetric operating system that allows multiple processors to execute simultaneously within the single, memory-resident image of the kernel. Because the kernel is multithreaded, it is fully preemptable, ensuring that high priority real-time threads can immediately preempt lower priority threads running in user or kernel mode. Extensive use of spin-locks rather than kernel preemption points means that kernel threads typically spin rather than blocking when operating on critical sections of code, thereby improving scalability.

Priority-Based Scheduling To ensure that real-time threads are never preempted by lower-priority threads, the Solaris Operating Environment provides a set of application-appropriate scheduling classes. Priorities within these classes are mapped to a set of global priorities (Figure 2). Time-share threads are scheduled in a round-robin fashion. System threads are given higher priority than time-share threads and are scheduled by the system. Realtime threads run at a higher priority still; different real-time priorities are assigned different (adjustable) time quanta. A priority inheritance protocol minimizes the effect of priority inversion where real-time threads contend with lower-priority threads for synchronous objects. Both kernel and user-level priority inheritance as well as user-level priority ceilings are provided.

6

Scalable Real-Time Computing in the Solaris Operating Environment

Fine-Grained Processor Control The Solaris Operating Environment allows threads and applications to be bound to specific individual processors. In addition, processors within a system can be grouped together as a processor set and dedicated to real-time tasks. The processor set can be sheltered from non-deterministic time-share and system processes as well as unbound interrupts for even greater real-time determinism and performance.

Time-Share and System Threads

CPU 1

Real-Time Threads and Bound Interrupts

Unbound Interrupts

CPU 1

CPU 1

CPU 1

Processor Set

FIGURE 2

Processor sets allow dedicated resources for real-time tasks

Additional processors can be added to the processor set as real-time application demand increases. This can be especially useful in environments such as weapons systems where the CPU power is often grossly underestimated for operation in a war-time environment. Processor sets provide the flexibility to easily and dynamically configure the system for different usage patterns before or during the critical time frame.

Precision Timers and Clocks The Solaris 8 Operating Environment provides sophisticated timing resources, including support for the network time protocol (NTP) as well as both UNIX and POSIX timers. High-resolution hardware timers are also supported. ®

Other Real-Time Related Features The Solaris Operating Environment provides a wealth of features that enhance real-time applications, including prioritized STREAMS, 64-bit and large file system support, TCP large windows, TCP selective acknowledgment, zero-copy TCP, and others.

A New Look at Real-Time Computing

7

CHAPTER

2

Real-Time Innovations in the Solaris Operating Environment The real-time capabilities in the Solaris Operating Environment are derived from the fundamental capabilities of its innovative architecture, including a fully preemptable multithreaded kernel, priority-based scheduler, and precision timers and clocks. Together, these and other features assure bounded response time and enable a wealth of real-time application possibilities.

Multithreaded Kernel The kernel of the Solaris Operating Environment is fully multithreaded — eliminating the need for numerous kernel preemption points. This innovative architecture presents considerable flexibility for structuring and deploying applications, and allows resources to be added where they are needed most.

A Scalable, Flexible Model Figure 1 schematically depicts the Solaris Operating Environment and represents the user, kernel, and hardware layers.

8

User Processes

User Layer

Daemon

Daemon

Kernel Layer

Hardware Layer Legend:

FIGURE 1

Application Thread

LWP

Kernel Thread

Processor

Execution components of the Solaris Operating Environment

The following execution components are illustrated: ■

Application threads represent a sequence of instructions within a user process. A user process is comprised of one or more application threads, each of which can execute independently or concurrently.

■

Lightweight Processes (LWPs) can be considered as “virtual” CPUs onto which application threads are scheduled (or multiplexed) by the threads library. Each LWP is associated with a kernel thread.

■

Kernel threads are the only entity to which the kernel has access for scheduling purposes. Each user process is associated with at least one kernel thread through an LWP. However, some kernel threads are not associated with a LWP, for instance, system daemons such as the page daemon, NFS daemon, and the clock thread require only the resources of a kernel thread.

Real-Time Innovations in the Solaris Operating Environment

9

Processor Binding and Processor Sets The two-level model depicted in Figure 1 was designed to make user-level threads quick to create and inexpensive in terms of resource consumption. For real-time applications, threads can be bound to LWPs through the threads API. In addition, both LWPs and processes can be bound to individual processors using the processor_bind(2) function and pbind(1M) system call. However, simply binding a process or LWP to a processor does not preclude other unbound LWPs from also running on the same processor. To address this issue, Sun provides processor sets.

Processor Sets First available in the Solaris 2.6 Operating Environment, a processor set consists of a number of processors and a corresponding number of LWPs which are bound to the processor set. Programmers can use pset_create(2) to create a processor set, pset_assign(2) to change the processors assigned to the set, and pset_bind(2) to bind one or more LWPs to a processor set. Processor sets may also be manipulated from the command line with psrset(1M). Processor sets have the following properties: ■

LWPs bound to a processor set will only run on the processors in the processor set. LWPs are restricted to being bound to a single processor set.

■

LWPs can be bound to individual processors within a processor set.

■

As with placing processes in the real-time scheduling class, processor sets can only be manipulated with root privileges.

■

Interrupts may still be delivered to processors assigned to a processor set unless sheltered as described below.

Interrupt Sheltering The Solaris 7 and 8 Operating Environments enable processors to be sheltered from unbound interrupts. At least one processor on the system must remain unsheltered to service the system clock interrupt. The psradm(1M) command can be used to shelter a single processor from unbound interrupts by placing it in the P_NOINTR state. The psrset(1M) command may be used to shelter all of the processors in the specified processor set from unbound interrupts. This technique can be useful for real-time applications since properly sheltered processors in a processor set will never run unbound interrupts. Bound interrupts (those associated with a bound processes or thread) are unaffected by the sheltered (P_NOINTR) state of a processor and execute as expected.

10

Scalable Real-Time Computing in the Solaris Operating Environment

Priority-Based Scheduling Traditional UNIX scheduling algorithms were originally designed to maximize system throughput in a multiuser, time-sharing environment. To provide equitable behavior, the priorities for resource-intensive jobs were lowered automatically. Though useful for general-purpose systems, this is undesirable behavior for a system hosting real-time applications.

Global Priority Model To enable deterministic behavior, the scheduler in the Solaris Operating Environment is designed to allow multiple scheduling classes, as illustrated in Figure 2. Level -10 59

160-169

Interrupts Level -1

Real-time 0 100 99 +60

System Time-share

Global Priority Range 60 59

-60 +60

Interactive

0

-60 FIGURE 2

Scheduling classes and global priority range

Real-Time Innovations in the Solaris Operating Environment

11

Within this scheduling model, each thread or process is given a priority within a scheduling class. Scheduling decisions are made based on global priority. Traditional means of adjusting priority (nice(1) values) affect priority within the processes’ scheduling class. The scheduling class and class-specific parameters can be manipulated with the priocntl(1M) command. The following classes are defined in the Solaris Operating Environment: ■

Time Share (TS) – Provides traditional UNIX process scheduling, assuring fairness and maximizing throughput (priorities 0-59)

■

Interactive (IA) – Provides responsiveness for user interaction by favoring processes with keyboard focus (for a system running a windowing system)

■

System (SYS) – Provides fixed-priority scheduling for kernel threads such as the pageout thread and RPC service threads (priorities 60-99)

■

Real Time (RT) – Provides fixed-priority preemptive scheduling for user processes (priorities 100-159)

As shown in Figure 2, interrupt threads are mapped above real-time priorities.

The Real-Time Scheduling Class The Solaris Operating Environment real-time scheduling class is compliant with POSIX 1003.1b. Users may specify either a first-in-first-out (FIFO) or round-robin scheduling policy. ■

First-in-first-out (FIFO) The FIFO scheduling policy implements fixed priority with pure priority preemption. The highest priority thread that is ready to run will execute until it either blocks or a higher priority process becomes runable. When choosing between two threads with equal priority, the scheduler selects the thread that was ready to run first (FIFO behavior).

■

Round-robin The round-robin scheduling policy adds an execution time quantum. The highestpriority, ready-to-run thread will run until it blocks, is preempted by a higher priority process, or its time quantum expires and a thread of equal or greater priority is ready to run. Different real-time priorities have different associated time quanta that are configurable via dispadmin (1M).

LWPs in the RT scheduling class will never be swapped out, but they may incur pagefaults (see Chapter 3 on memory locking). Only superuser (root) may put LWPs in the RT scheduling class, though the LWPs themselves need not run as root. The configuring of RT priority to global priority is done via rt_dptbl(4).

12

Scalable Real-Time Computing in the Solaris Operating Environment

Priority Inversion Priority inversion occurs when a high-priority thread blocks on a resource that is held by a lower-priority thread. A runable thread with a priority between the highand low-priority threads creates a priority inversion because it can receive processor resources ahead of the high-priority thread. To avoid priority inversions with kernel synchronization primitives, the Solaris Operating Environment employs a transient priority inheritance protocol. The protocol enables the low-priority thread holding the resource to “inherit” the higher priority of the blocked high-priority thread. This approach gives the blocking lowpriority thread the CPU resources it needs to complete its task as soon as possible so that it can release the synchronization primitive. Upon completion, all threads are returned to their respective priorities by the kernel. Kernel priority inheritance has been provided in the Solaris Operating Environment kernel since its inception. The Solaris 8 Operating Environment extends this concept to the user level, providing both priority inheritance and priority ceilings at the user level. Priority ceilings place a limit on how high the priority for a user process can be raised.

Interrupts The Solaris Operating Environment provides fifteen interrupt levels (level-1 through level-15). Interrupts at or below level-10 are handled as threads and thus may block. Interrupt threads occupy the highest range of global priority, with level-n interrupt threads at a higher priority than level-(n-1). Most devices interrupt below level-10; the system clock executes at level-10. Handling interrupts as threads allows most device drivers to protect critical sections with synchronization primitives without manipulating programmable interrupt levels. Interrupt threads are processor-bound — when an interrupt thread at a certain level blocks, the processor will run other threads but will not lower the programmable interrupt level below the level of that thread.

Real-Time Innovations in the Solaris Operating Environment

13

Precision Clocks and Timers Precision timing and synchronization is key for real-time applications. Clocks provide a mechanism for obtaining timestamps for events, or for measuring the wall-clock time that has elapsed between examination points in a program. Interval timers work by sending signals to requesting processes after a specified period of time has elapsed.

Standard Clocks and Timers The Solaris Operating Environment has always featured traditional BSD-style UNIX interval timers (manipulated by setitimer(2), and getitimer(2)). Support for POSIX.1b timers was included first in the Solaris 2.6 Operating Environment (manipulated by timer_create(3RT) and timer_settime(3RT)). POSIX timers (defined as “the amount of time since the Epoch”) are based on the clock type CLOCK_REALTIME and are available on a per-process basis. Per-thread timers can be obtained by using signal masks. POSIX and BSD timers as well as poll(2) time-outs are dispatched by the system clock. By default, the system clock executes at 100 Hz yielding a 10-millisecond timer granularity.

Configurable Hertz For many real-time applications, the standard system clock resolution of 10 milliseconds (100 Hz) is too low. Version 2.6 of the Solaris Operating Environment addresses this issue by allowing the system clock rate to be configured. Setting the variable “hires_tick” to “1” in /etc/system has the effect of setting the system clock to execute at 1000 Hz — yielding a one-millisecond granularity. For some applications, a one-millisecond timer resolution is still too coarse. However, though higher values of hires_tick are possible, they are not recommended or supported. Tuning the system clock too high can result in the system running nothing but the clock code with undesirable performance side-effects. Additionally, using standard POSIX timers with or without configurable hertz is often not ideal for real-time applications needing interval timers that are not evenly divisible by the system heart-beat.

14

Scalable Real-Time Computing in the Solaris Operating Environment

High-Resolution Hardware-Based Timers The Solaris 8 Operating Environment offers a superior solution to timer granularity by providing access to high-resolution timing resources available on most newer processors. A new POSIX clock type (CLOCK_HIGHRES) is now available to access high-resolution, hardware-based timers. CLOCK_HIGHRES timers do not rely on the system clock for dispatch but rather deal directly with the per-processor hardware timer source (typically the processor where the real-time process is running). Because these timers are not subject to adjustments of the system clock (for clock drift or external updates), they offer reliable, non-adjusted, interval-based timing. Timer resolution is as fine-grained as the platform can provide (up to one nanosecond). The actual hardware timer source used is transparent to the application and may change over the life of the timer. Furthermore, this approach helps ensure that timer resolution for real-time applications will scale as processors become faster. In practice, the processor binding of a CLOCK_HIGHRES timer shadows that of its creating LWP. If a processor is assigned to a processor set, only CLOCK_HIGHRES timers for LWPs bound to that set will use the processor’s timer source. By default, CLOCK_HIGHRES timers may only be created by root, though this is tunable by setting a variable in /etc/system. CLOCK_HIGHRES timers for bound LWPs are considered to be bound interrupts and are unaffected by setting P_NOINTR. The clock_getres(3RT) call is used to determine the platform’s CLOCK_HIGHRES resolution.

CLOCK_HIGHRES on UltraSPARC On UltraSPARC systems, CLOCK_HIGHRES is driven with the per-processor TICK_COMPARE register, which counts the number of hardware clock ticks since power-on. The CLOCK_HIGHRES resolution on an UltraSPARC system is thus the reciprocal of the processor clock rate. For example, a 250-MHz UltraSPARC processor has a CLOCK_HIGHRES granularity of four nanoseconds. On platforms (such as UltraSPARC) that provide extremely fine-grained CLOCK_HIGHRES resolution, timers with small intervals can result in very high interrupt rates that will swamp the processor.

CLOCK_HIGHRES on Other Platforms On platforms other than UltraSPARC processors, the hardware timer source is interval-based (typically one hertz). Setting “hires_tick” in /etc/system is still important for real-time applications that need high resolution timers on nonUltraSPARC platforms.

Real-Time Innovations in the Solaris Operating Environment

15

Timestamps Applications requiring time-of-day timestamps should use gettimeofday(3C). Applications that need fast, nanosecond-resolution timestamps should always use gethrtime(3C). As mentioned, gethrtime(3C) isn’t correlated to the time of day, and is not subject to resetting or adjustment by way of adjtime(2) or settimeofday(3C).

Network Time Protocol (NTP) In addition to these timing resources, the Solaris 8 Operating Environment provides support for the NTP, which provides both time and/or network clock synchronization for use in distributed computing environments. The new support for NTP provides increased performance and precision over publicly-available versions.

Features for Improved Determinism Virtual memory, dynamic libraries, and high-level networking provide significant flexibility in a general-purpose operating system. In a real-time system, these features can introduce small, unpredictable timing variances — jitter — and reduce determinism at run-time. The Solaris Operating Environment provides mechanisms and improvements that help real-time applications gain the predictability they need.

Memory Locking In a virtual memory system, any memory reference can potentially cause a page fault. The time required to bring the referenced data into physical memory from disk can cause a real-time application to lose its determinism. The Solaris Operating Environment addresses this problem by allowing the locking of a process’ pages into memory using mlock(3C) or mlockall(3C). In any real-time application, all translations should be faulted in and locked down before time-critical code is executed. All current and future faulted translations may be locked by specifying the MCL_CURRENT and MCL_FUTURE flags respectively to mlockall(3C). Locked translations can be unlocked with munlock(3C). Like placing an application in the real-time scheduling class, root access is required to lock pages into memory.

16

Scalable Real-Time Computing in the Solaris Operating Environment

Early Binding By default, linking of dynamic libraries in the Solaris Operating Environment is done on an as-needed basis. The run-time binding for a dynamically linked function isn’t determined until its first invocation. Though flexible, this behavior can induce indeterminism and unpredictable jitter in the timing of a real-time application. To avoid jitter, the Solaris Operating Environment provides for early binding of dynamic libraries. By setting the LD_BIND_NOW environment variable to “1”, libraries are bound at application startup time. Using early binding together with memory locking is a very effective approach to avoiding jitter.

Priority STREAMS Previous to the Solaris 8 Operating Environment, the STREAMS framework (used for networking) did not take the priority of an executing thread into account. Under certain conditions, STREAMS could promote a lower priority thread above a higher priority thread, effectively causing a priority inversion. Sun has made significant improvements to the STREAMS package in the Solaris 8 Operating Environment so that high-priority threads (interrupt threads and threads in the real-time scheduling class) are not usurped to perform lower-priority work. These improvements include: ■

Allowing service processing to be done only on the stream associated with the thread that generated the pending STREAMS operation

■

Allowing a stream to take on the priority of the thread that has sourced a message to it, or the current priority of the stream, which ever is higher

■

Changing the streams processing scheduler to support priorities that are based on a thread/stream, and providing for the promotion of a lower-priority thread to run at least at the priority by which a message was generated

While these changes do not put bounds on response time for STREAMS, the framework makes all efforts to process the real-time or time-critical task as quickly as possible. By implementing this as the default behavior for STREAMS, all system users receive the benefit of more predictable behavior.

Real-Time Innovations in the Solaris Operating Environment

17

CHAPTER

3

A Compelling Development and Run-Time Environment Beyond the fundamental mechanics required for time-critical applications, real-time developers need access to a full-featured operating system. The Solaris Operating Environment provides this functionality along with flexible application choices and powerful development tools.

Networking and I/O Technology for Real-Time Applications By their nature, real-time applications often need to capture or transport significant amounts of data quickly. I/O and networking are especially important to real-time applications. The Solaris Operating Environment provides significant capabilities in these areas to help applications transmit and operate efficiently on large volumes of data.

64-Bit Large Files The 64-bit capabilities of the Solaris Operating Environment enable operations on large files — files that are larger than or equal to two gigabytes in size. Large file semantics are supported on the UFS, NFS , and CacheFS file systems. Specifically, applications are able to create and access files up to: ™

18

™

■

One terabyte on UFS mounted file systems

■

The limit of the NFS server for NFS and CacheFS file systems

Data types and file I/O interfaces (most with suffixes of 64 or 64_t) are provided to operate on large files.

Raw and UFS Direct I/O Applications often need to deal with data outside of traditional buffering and file system constraints. The Solaris Operating Environment offers the ability to write directly to raw devices that do not have a file system. Raw I/O can dramatically increase throughput to disk devices. Additionally, direct I/O enables programmers to place data directly on disk in a UFS file system without buffering the data in memory. Bulk I/O operations use larger buffer sizes to transfer large files (files larger than physical memory), such as those required for downloading satellite information.

Kernel Sockets The Solaris 8 Operating Environment kernel sockets implementation provides improved compatibility with SunOS 4.x and BSD sockets. The implementation provides higher socket performance while preserving the high performance of TCP/IP based on STREAMS. ™

TCP Large Windows TCP large windows provides the support described in RFC 1323, which is designed to improve performance over large bandwidth networks such as ATM or high-delay networks such satellite links. TCP large windows support works by expanding the amount of data that can be outstanding in a TCP session from 65,535 bytes to approximately one gigabyte. To prevent truncation, both ends of a TCP connection must support TCP large windows.

A Compelling Development and Run-Time Environment

19

TCP Selective Acknowledgment The Solaris 8 Operating Environment provides support for selective acknowledgment as described in RFC 2018. Selective acknowledgment is a strategy that can improve throughput in the face of multiple dropped segments within a TCP window. Using this approach, the receiving system can inform the sender about all segments that have arrived successfully, so the sender need retransmit only the segments that have actually been lost. As with TCP large windows, both sides of a TCP connection must support TCP selective acknowledgment.

Zero Copy TCP/Hardware Checksum Zero copy TCP/hardware checksum functionality can be used to increase TCP throughput in certain situations by utilizing hardware checksum logic in the network interface controller when available. Currently, the only network interface that contains hardware checksum logic is the SunATM 622 interface. ™

Source Compatibility One of the more powerful aspects of real-time application development on the Solaris Operating Environment is that it offers a wide range of solutions and technologies while maintaining a single source tree. Not only does this approach save on development costs but it makes it easy to retarget applications to new hardware platforms and add new functionality to existing applications. In particular, developers can choose from: ■

UltraSPARC and Intel processors Full source compatibility means that developers can use the Solaris 8 Operating Environment to support both UltraSPARC and Intel platforms in a single source tree – the same source code can be simply recompiled to address different platforms.

■

32-bit or 64-bit applications The Solaris 8 Operating Environment features a full 64-bit operating system that can also run 32-bit applications. Both 64- and 32-bit support is provided on UltraSPARC platforms. 32-bit support is provided on Intel platforms with 64-bit Intel support on new Itanium-based platforms.

20

Scalable Real-Time Computing in the Solaris Operating Environment

■

Real-time and time-share applications Both real-time and time-share applications run under the Solaris Operating Environment together. Applications can even be mixed-mode – possessing both time-share and real-time threads. This flexibility enables components like GUIs to run as time-share threads with real-time threads performing time-critical operations. System resources are concentrated where they are needed for determinism.

■

C2 or B1+ security Developers can choose from C2 or B1+ security. B1+ security is only available today in Trusted Solaris software, a DoD-specific product, and typically requires a separate development effort. Role-based access control (RBAC) is also included in the Solaris 8 Operating Environment. ™

■

Quality-of-service tools Several tools are available to provide increasing levels of quality of service under the Solaris Operating Environment. Solaris Resource Manager software enables administrators to guarantee that certain percentages of computational resources are available to applications, users, and groups of users. Copackaged with Solaris Resource Manager, Solaris Bandwidth Manager enables administrators to control the bandwidth assigned to particular applications, users, and departments that share the same intranet or Internet link. ™

™

Powerful Development Tools Application development in most real-time operating systems is notoriously difficult and inflexible. In contrast, the Solaris Operating Environment features a powerful development environment for native, real-time applications that gives developers a wealth of choices with extensive tool sets from Sun and its associates.

Debugging — Trace Neutral Format (TNF) Trace neutral format (TNF) support in the Solaris Operating Environment kernel provides real-time application developers with visibility into the operation and data structures of the kernel through the prex(1) command. Kernel trace points enable developers to debug real-time applications by understanding the status of kernel variables, synchronization objects, and other information.

A Compelling Development and Run-Time Environment

21

™

Sun WorkShop Products Successful application development requires that programmers have highperformance compilers and tools. The Sun WorkShop family of products includes highly optimizing, automatically parallelizing compilers with versions supporting the Fortran, C, and C++ programming languages. Libraries of highly optimized routines are included along with tools to help analyze and tune code for additional performance. ™

Sun WorkShop features include: ■

Integrated programming environment

■

Motif user interface, providing a standard look and feel

■

Tight, editor-centric tool integration

■

Hyperlinks, enabling easy tool navigation

■

Multiprocessing, multithreaded development tools

■

Distributed and parallel make utilities

■

Incremental linker for faster builds

■

Fix and Continue, enabling defects to be found and fixed quickly

■

AppGuru, enabling very fast application development for C++

■

New version of Rogue Wave Tools.h++ 7.0 class library

■

Motif, Windows, and Java GUI Builder for cross-platform development

■

Sun Visual WorkShop software, enabling quick and easy GUI development

■

GUI capture and testing, providing reverse engineering capabilities

■

Three-dimensional data visualizer, speeding debugging of complex arrays

■

WorkSets and PickLists, facilitating quick access to work sessions

■

Sun WorkShop TeamWare for source code and configuration management

™

™

™

Of particular interest to developers is the ability of Sun WorkShop products to perform advanced optimizations that can speed applications performance including:

22

■

Instruction scheduling – Arrange the order in which instructions are executed and make optimal use of available machine resources

■

Profile feedback – Obtain frequency information for application optimizations such as code motion and inlining

■

Loop parallelization – Rearrange loop code so that multiple processors may be work in parallel to complete the loop

■

Cache blocking – Rearrange loop code to maximize the use of the processor cache

■

Loop inversion – To reverse the order of nested loops to gain the advantages of improved loop parallelization or better cache blocking

Scalable Real-Time Computing in the Solaris Operating Environment

Sun WorkShop compilers support both traditional and hybrid versions of the SPARC Version 9 architecture. Full 64-bit computing is available with SPARC V9 support. A hybrid version, called V8+, precludes the use of all V9 64-bit addressing instructions, ensuring 32-bit compatibility with existing versions of Solaris software and other existing applications, while still allowing access to most UltraSPARC system advanced capabilities, including the VIS instruction set. ™

Java Applications Development A discussion of software development would not be complete without mentioning Java technology. Taking the industry by storm, Java technology promises true platform-independent software development for a large number of applications. Software developers instantly recognized the potential of Java applications, with thousands of firms currently developing or planning to develop Java technologybased products. As the creator of Java, Sun offers software developers with a comprehensive product line designed to streamline development using Java technology. The object-oriented Java platform delivers the benefits of reusable code, reduced cost of ownership, and broad integration without the complex, heavyweight object housekeeping process required by other object-oriented development models. Sun’s family of Java APIs and development products, including Java WorkShop , Java 2 SDK, and Java Studio software, empower developers to create an new class of applications that truly enable network-based computing. Sun Ultra workstations are ideal platforms for these tools, permitting the development of both client and server components of Java solutions. ™

™

™

™

The Java 2 SDK is bundled with the Solaris 8 Operating Environment, and provides both essential development tools required for creating applications in the Java language and a high-performance, scalable runtime environment.

Real-Time Java Implementation A full real-time Java implementation will require a real-time Java virtual machine. The on-going Java Community Process is currently defining an open specification for a real-time Java virtual machine.

A Compelling Development and Run-Time Environment

23

CHAPTER

4

Summary: Sun’s Real-Time Vision The Solaris Operating Environment is a real-time operating system, in both architecture and implementation. Sun’s flexible and powerful multiprocessor systems can achieve bounded response times and are especially appropriate for compute- or data-intensive real-time applications. The ability to combine scalable real-time performance with a full spectrum of services represents a significant opportunity for real-time application developers. Real-time computing is an area of focus for the Solaris Operating Environment, and Sun will continue to provide cost-effective scalable real-time infrastructure support with particular focus for ongoing work in the following areas:

24

■

Interrupt routing to a particular processor

■

In-kernel tracing

■

Cyclics – High-resolution timers

■

802.1p – Deterministic networking

■

Quality of service (QoS) API

■

Distributed real-time capabilities

CHAPTER

5

Appendix

POSIX Real-time Extensions POSIX 1003.1b (formerly known as 1003.4) provides requirements for real-time operating environments. POSIX 1003.1c (formerly known as 1003.4a) provides interfaces and functionality to support multiple flows of control (threads) within a process. Any implementation can claim POSIX conformance if it provides library bindings for the defined interfaces — regardless of whether the library bindings are backed up by actual implementations of the appropriate functionality. The library bindings can return an error code (ENOTSUPP) if the functionality is not implemented. The Solaris Operating Environment implements virtually all POSIX 1003.1b and POSIX 1003.1c functionality as shown in the following tables.

25

Supported POSIX.1b Extensions POSIX Feature

Supported? 1

Notes

Asynchronous I/O

Yes

Mapped Files

Yes

mmap(2), munmap(2)

Memory Locking

Yes

mlock(3C), munlock(3C), memcntl(2)

Range of Memory Locking

Yes

mlockall(3C), munlockall(3C)

Memory Protection

Yes

mprotect(2)

1

aioread/write(3), aiocancel(3),...

IPC message queues

Message Passing

Yes

Prioritized I/O

No

I/O priority based on process priority

Priority Scheduling

Yes

Global priorities plus a “nice” value

Real-time Signals

Yes

SIGRTMIN=37, SIGRTMAX=44

1

Semaphores

Yes

File Synchronization

Yes

Shared Memory Objects

Yes

Synchronized I/O

Yes

Timers

Yes

1

1

IPC Semaphores fsync(3C), fdatasync(3R) IPC Shared Memory msynch(3C), fcntl(2),... CLOCK_HIGHRES

1. Solaris-specific feature available

TABLE 1

26

POSIX 1003.1b extensions supported in Solaris Operating Environment Version 2.6 or later

Scalable Real-Time Computing in the Solaris Operating Environment

Supported POSIX.1c Extensions POSIX Feature

OS Version 2.6

7

8

_POSIX_THREADS

Yes

Yes

Yes

_POSIX_THREAD_ATTR_STACKSIZE

Yes

Yes

Yes

_POSIX_THREAD_ATTR_STACKADDR

No

Yes

Yes

_POSIX_THREAD_ATTR_PRIORITY_SCHEDULING

No

Yes

Yes

_POSIX_THREAD_ATTR_PRIO_INHERIT

No

No

Yes

_POSIX_THREAD_ATTR_PRIO_PROTECT

No

No

Yes

_POSIX_THREAD_ATTR_PROCESS_SHARED

Yes

Yes

Yes

_POSIX_THREAD_SAFE_FUNCTIONS

Yes

Yes

Yes

TABLE 2

POSIX 1003.1c thread extensions supported in the Solaris Operating Environment

Appendix

27

Sun Microsystems, Inc. 901 San Antonio Road Palo Alto, CA 94303 1 (800) 786.7638 1.512.434.1511 http://www.sun.com/solaris/

February 2000