CleanCapture™SiP-1280 Digital Image Processor A No-Compromise Approach to Image Quality and Performance

i

Copyright Copyright © 2001-2004, NuCORE Technology Inc. – All Rights Reserved

Disclaimer The information in this document is preliminary and is subject to change without notice. Information in this document is provided in connection with NuCORE products. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted by this document. Except as provided in NuCORE’s Terms and Conditions of Sale for such products, NuCORE assumes no liability whatsoever, NuCORE disclaims any express or implied warranty, relating to sale and/or use of NuCORE products including liability or warranties relating to fitness for a particular purpose, merchantability, or infringement of any patent, copyright or other intellectual property right. NuCORE products are not intended for use in medical life saving, or life sustaining applications. NuCORE may make changes to specifications and product descriptions at any time, without notice. Designs must not rely on the absence or characteristics of any features or instructions marked “reserved” or “undefined.” NuCORE reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The SiP-1280 may contain design defects or errors known as errata that may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local NuCORE sales office or your distributor to obtain the latest specifications and before placing your product order.

Trademarks NuCORE Technology, NDX Technology, SiP Technology, CleanCapture, and InfiniteBurst are trademarks of NuCORE Technology. Third Party brands and names are the property of their respective owners. Copies of documents that have an ordering number and are referenced in this document, or other NuCORE literature may be obtained by contacting NuCORE Technology.

Document Number 1280WP-rev1

Sales Offices USA

Japan

NuCORE Technology Inc.

NuCORE Technology Co., Ltd.

1380 Bordeaux Drive Sunnyvale, CA 94089

Nisso No. 12 Bldg., 2F 3-6-12 Shin-Yokohama, Kohoku-Ku Yokohama, Japan 222-0033

Phone: (408) 907-7100 Fax: (408) 745-1860

Phone:81-45-471-8745 Fax: 81-45-471-8748

Email: [email protected]

Email: [email protected]

ii

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

Until now, discussions surrounding digital imaging technology have concentrated simply on the number of MegaPixels. NuCORE Technology is now refocusing these discussions on the issue of image quality and performance with its CleanCapture™ Technology and InfiniteBurst™ Capture. The CleanCapture™ Technology dramatically minimizes color noise and artifacts in image capturing and processing which will allow digital cameras to capture truer representation of the actual subject being photographed. On the performance side, InfiniteBurst™ Capture is an industry first which enables unlimited burst capture of high-resolution (e.g. 5 MegaPixels) still images at an amazing 3-4 frames/second until the flash memory is full – other solutions typically run at half the frame rate and stall the burst capturing after several images for process and storing to external media.

Overview The CleanCapture™ SiP-1280 is NuCORE’s third generation digital image processor for digital cameras and digital video camcorders. Coupled with NuCORE’s NDX-1260 Analog Image Processor delivers the highest quality images which are free of noise and artifacts. The design philosophy of the SiP-1280 is to use high precision no compromise image algorithms. The algorithms are embedded into a flexible hardwired image pipeline (see yellow blocks of the first diagram of the Internal Architecture section) to deliver an unprecedented 81 MegaPixels/second. For the end-user, this architecture directly translates into the ability to do infinite high-resolution burst capture and HDTV movie capture and playback. In addition to the breakthrough in quality and performance, NuCORE provides a platform solution with a suite of developer tools. This includes a comprehensive API, which dramatically reduces time-to-market, and a real-time camera tuning application that allows OEMs to quickly customize and optimize performance. In addition to CleanCapture™ Technology and InfiniteBurst™ Capture, NuCORE has also added other ground-breaking new features to the SiP-1280: •

HDTV movie capture and playback – As mentioned above, the SiP-1280 has the amazing ability to capture and playback HDTV movies at 1280x720 (720p resolution) at fifteen frames/second using MJPEG.

•

DirectView™-HD – direct connection to HDTV monitors to display still images at resolutions up to 1920x1080 (1080i), which is over two MegaPixels – this is over six times the resolution on standard NTSC resolution. This also allows HDTV movie playback to be viewed at the full 720p resolution.

•

Integrated high-speed USB 2.0 – allows ultra fast transfer of still images and video to a PC – compared to the standard USB 1.1 found in most cameras today, a hundred 5 Megapixel still images are transferred in seconds and not minutes.

•

Wireless LAN support – The SiP-1280 SD flash memory interface also includes the support for SDIO which enables a camera to use a Wireless LAN SD module to do remote file transfer to a PC or printer.

© 2004 NuCORE Technology Inc Page 1

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance •

Digital LCD interface – a significant power savings can be obtained by using a digital connection (instead of analog) to next generation digital LCD panels.

•

PictBridge Support – an industry standard for allowing a direct connection between a camera and a printer – a PC is not required to print pictures.

The SiP-1280 is uniquely positioned to support a new generation of hybrid cameras – a cross between digital cameras and digital camcorders. These devices will need high capacity storage; either flash memory, hard drives, or optical storage, all of which are directly support by the SiP-1280. In addition, hybrids will also need MPEG2 or MPEG4 video compression – the SiP-1280 has a digital video port with allows for a direct connection to these types of video codec. For fast time to market, the SiP1280 evaluation kit includes a NEC uPD61151 MPEG2 codec with its functionality completely integrated into the NuCORE software API. InfiniteBurstTMCapture InfiniteBurst™ Capture is an industry first which enables unlimited burst capture of highresolution (e.g. 5 MegaPixels) still images at an amazing 3-4 frames/second until the flash memory is full – other solutions typically run at half the frame rate and stall the burst capturing after several images for process and storing to external media. Below is a diagram which shows a typical architecture which can’t support infinite burst capture because the image processing steps are all serialized. Once the internal buffer is full (steps 1, 2 and 3), the camera can not take any more pictures and there is a significant stall while the internal buffer is emptied to the flash memory (steps 4 and 5). Also if the captured image needs to be displayed on a LCD, this will typically reduce the overall frame rate of the burst capture: SDRAM / SRAM Temporary Buffer size limit

4

3

Simultaneous LCD Display Limitation

1

•Capture image from CCD •CCD Speed limits burst capture speed

2

•Store image into temporary buffer until it is filled, •camera then stalls to process burst image sequence

3 4

Final Storage Write Bandwidth Limitation

5

DIP Zoom JPEG OSD

1

Image Processor Limitation 2

•LCD Display, if on, often diminishes capture burst performance •Image processor operations from buffer AFTER burst capture is complete •DIP, Zoom / resizing, JPEG, OSD

5

•Store to flash memory card

© 2004 NuCORE Technology Inc Page 2

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance As shown in the diagram below, the SiP-1280’s high-performance architecture includes a pipelined DMA which can simultaneously move data through the image processing pipeline to the internal buffer and LCD (step 1) and from the internal buffer to flash memory (step 2).This pipelined architecture, coupled with the ultra fast 81 Megapixel/sec image pipeline is what allows the SiP-1280 to do infinite burst captures at such high frame rates: CCD Speed Limitation

Concurrent IPU Multi-Stage Pipelined DMA

DIP Zoom JPEG OSD

1

SDRAM / SRAM Temporary no Buffer limitation

1•

2

Capture image from CCD •

Final Storage

2 Simultaneous LCD No Limitation Concurrent Operation

CCD Speed limits burst capture speed

Simultaneous LCD display with no capture performance degradation Pipelined Processor with dedicated DMA processes final store enabling infinite burst capture • •

DIP, Zoom / resizing, JPEG, OSD No stalling – Camera continues to capture infinitely until removable memory card is filled

Simultaneous Store to flash memory card

From an end-user perspective the result is being able to take a large sequence of highresolution action pictures and the later choose the one that really captures the moment – very similar to what a $5,000 and up film SLR camera with a motor drive provides to professional sports photographers. An example of that is the baseball sequence below:

© 2004 NuCORE Technology Inc Page 3

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance From the sequence you can now choose the best action shots:

InfiniteBurst™ Capture is an industry first which enables unlimited burst capture of high-resolution still images at an amazing 3-4 SiP-1280 Differentiator → frames/second until the flash memory is full. Go to www.nucoretech.com for a flash demo of InfiniteBurst™ Capture.

© 2004 NuCORE Technology Inc Page 4

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The SiP-1280 Internal Architecture Below is a block diagram of the CleanCapture™ SiP-1280, showing all the major components of the hardwired image pipeline. Each block’s functionality is described in detail along with any key differentiators. After reviewing each block it should be obvious that NuCORE has gone the extra mile to provide a no-compromise world class Image Processor which will take digital cameras and digital video camcorders to the next level of image quality and performance. NTSC or Y/C

AFE interface

Frontend resize

Bayer to RGB CYMG to RGB AE-AF-AWB Statistics

Color Correction

TV

Digital White Balance

Pixel Error Correction

HDTV

Component Video

Edge Enhancement

Digital LCD

CIR656

SiP1280 IPU

Chroma Suppression

Y-Gamma Compensation

Skin Tone Detection

YCrCb to RGB

Down Sample Filter

Digital Effects RGB to YCrCb

ARM9 Embedded Processor

Backend resize

Digital Zoom Engine

JPEG MJPEG Engine

YUV Conversion

4:4:4 ' 4:2:2

Digital I/O

CleanCaptureTM

RGB Gamma Correction

4:2:2

MPEG2 MPEG4 Digital Video CODEC

High Speed USB 2.0 SDIO SD Memory WLAN Card Card Compact Flash

OSD Engine

Hard Drive CDR AC-97 Audio Codec

© 2004 NuCORE Technology Inc Page 5

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

Bayer to RGB and CYMG to RGB Block Image sensors record only the brightness of light ranging from pure white to pure black:

How then, do sensors capture colors when all they can do is record light intensity? The trick is to use differently colored filters on the sensor's photosites to separate the colors of the light reflected from a scene. Sensors in most DSCs (Digital Still Cameras) have a Bayer filter that uses Red, Green, and Blue (RGB) color scheme, but the sensors in most DVCs (Digital Video Camcorders) use a Complementary filter that uses Cyan, Yellow, Magenta, and Green (CYMG) colors. The CYMG color scheme is more sensitive to low light, which works better in DVCs since they don’t have flashes like DSCs. The SiP-1280 supports both sensor types. The market is quickly moving to hybrid devices that have both DSC and DVC functions. The SiP-1280 is one of the only image SiP-1280 Differentiator → processors that supports both Bayer RGB sensors for high-resolution still images and the Complementary CYMG color scheme for video. Let’s look in more detail at the process of converting Bayer to RGB. A Bayer mosaic filter color is placed over individual photosites so that each can capture only one of the three colors:

As seen in the above picture, the Bayer pattern is designed so that one-fourth of the photo is captured in red light, one-fourth in blue, and one-half in green. Green is given more emphasis because it contains more light intensity, which is more noticeable to the human eye. To generate the final full-color image, a process called interpolation is used. Interpolation uses the colors of neighboring pixels to calculate the two colors a photosite didn't record. By combining these two interpolated colors with the color measured by the site directly, the original color of every pixel is calculated. This step is compute-intensive since comparisons with as many as four neighboring pixels are required to perform this process properly.

© 2004 NuCORE Technology Inc Page 6

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

The above diagram is the conceptual idea of how the center pixel is about to be interpolated from the colors recorded by the four surrounding pixels. The SiP-1280’s actual interpolation is more complicated and proprietary. It has a dedicated hardware block, which uses complex algorithms to convert the Bayer pattern from the image sensor to standard RGB. This conversion is one of the first things that occurs in the image processing pipeline and the precision of this conversion has a tremendous effect on the final quality of the image. The SiP-1280 implements Bayer-to-RGB conversion using a proprietary algorithm that allows very precise control in converting pixels from the Bayer pattern to the RGB color space. The algorithms SiP-1280 Differentiator → perform high precision calculations which have more decimal places to minimize rounding error. This produces sharper images with finer granularity and less false colors. Pixel Error Correction Data from the CCD image sensor may have missing pixel data, caused by a defective pixel in the sensor, which is typically “stuck” in the open collector position, called a white or hot pixel. These missing pixels can be recovered by looking at the pixels around the bad pixel and approximating its value. Most pixel error correction algorithms only use the nearest neighbors in the calculation, but NuCORE uses a proprietary two-dimensional error correction algorithm that uses more than the nearest neighbors to produce a more accurate approximation of the bad pixel. Error correction is implemented for Bayer RGB and Complementary sensors. Below is a three-way comparison of the original uncorrected bad pixels; the typical pixel error correction which leaves noticeable artifacts; and NuCORE’s superior error correction which completely removes the bad pixels:

Error pixel (Uncorrected)

Typical pixel error correction

NuCORE pixel error correction

© 2004 NuCORE Technology Inc Page 7

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The SiP-1280 uses a more sophisticated two-dimensional error SiP-1280 Differentiator → correction algorithm for improved image quality.

Digital White Balance White balance is another form of color correction to deal with differing lighting conditions. Normally our eyes compensate for different lighting conditions, but to correct other colors cast by the same light with a digital camera, the camera has to find the "white point" to make sure that a white object will appear white. Most digital cameras feature automatic white balance. This means that the camera looks at the brightest pixels to determine the white pixels and calculates the best-fit white balance, however these systems are often fooled (especially if taking a photograph dominated by one color, say green). Most cameras also allow you to override the automatic white balance by choosing a white balance manually, typically sunlight, cloudy, fluorescent, incandescent, etc. The SiP-1280 provides a color-by-color programmable fine white balance control with ultra high precision steps. For the best image quality NuCORE recommends that the coarse white balance control SiP-1280 Differentiator → be performed with the CleanCapture™ NDX-1260’s analog white balance and the fine adjustment be performed with the SiP-1280’s digital white balance. AF-AE-AWB Statistic Have you ever missed that split second shot because your camera was trying to focus? Or have you ever received your prints back from the lab with nicely exposed backgrounds and dark subjects? If so, then you know the importance of having fast and accurate auto focusing and intelligent, accurate metering in your camera. This hardware block is responsible for providing the information or statistics for Auto Focus (AF), Auto Exposure (AE), and Auto White Balance (AWB), all requirements for fully automated consumer “point and shoot style DSCs. Auto Focus (AF) is a critical metric of camera performance, as it directly relates on how quickly one can take a picture. In addition, AF must be accurate to arrive at a crisp focus. Due to the complexity and criticality of AF, every camera manufacturer has its own color science on the best way to perform this operation. Thus to be effective at AF, the SiP1280 must provide an infrastructure for the camera manufacturer that is flexible, fast, and accurate. Most digital cameras derive AF information by analyzing the high frequency content in part of the image. The more high frequency content in that part of the image, the more in focus the image is. In the SiP-1280 there is a series of two high pass filters to process the © 2004 NuCORE Technology Inc Page 8

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance incoming data for each part of the image that is being analyzed. In addition, the SiP1280 is very fast at processing AF data due to a dedicated hardware engine that quickly analyzes AF data in each frame of information. The AF information is also moved to main memory via Direct Memory Access (DMA) to minimize the loading on the embedded CPU while processing images. Thus, in draft or preview mode the AF circuits can fully analyze the AF data at 30 fps and seamlessly provide this information to the embedded CPU for use in determining AF motor control. SiP-1280 Differentiator →

The SiP-1280 has fast and efficient hardware based circuitry to process AF data in real-time at 30 fps.

Another key attribute of a superior AF system is its capability to be tuned to meet the needs of each camera manufacturer. As noted previously, each manufacturer has their own “secret sauce”, and the SiP-1280 must be flexible to meet each manufacturer’s needs. The SiP-1280 provides that flexibility in a number of ways. The first major component of AF flexibility is the capability for the manufacturer to select the quantity, placement, and size of the “windows” where the AF information to be analyzed. As shown in the figure below, the total number of AF windows may be adjusted from a minimum of three to a maximum of nine. In addition, the size of each window and the position of each column or row of windows can be adjusted. This provides flexibility for the camera manufacturer to customize placement of the windows.

1 row / 3 windows mode

3 rows / 3 windows mode Whole Image

Whole Image

0

1

2

0

1

2

3

4

5

6

7

8

© 2004 NuCORE Technology Inc Page 9

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The second major component of the AF system’s flexibility is that it has a programmable AF filter to allow optimization for a particular camera’s CCD geometry, optical filter, and optical zoom.

SiP-1280 Differentiator →

In the SiP-1280, the AF measuring “windows”, and the way those windows are analyzed, are exceptionally flexible to allow each OEM to tune it for its particular AF scheme. In addition, all AF statistics gathering is hardware based for high performance focusing.

Auto Exposure (AE) is another critical function in any camera. For example, when taking an indoor picture of a family member against a window, the camera must accurately estimate that the person in the window is what you are exposing for, and that it is okay for the background outside the window to be overexposed. Alternately, when taking a scenic picture, the camera must correctly determine that the overly dark branch in the foreground can be underexposed, and that the key part of the picture to be properly exposed is the larger scene beyond the branch. To perform this AE function accurately, the camera must use advanced technology and algorithms to arrive at the optimal solution for each image captured. Early cameras used a technique that relies on center weighted metering. In this design, a single sensor detected how much light was falling on the image. This sensor was most sensitive to light in the center of the image, with less sensitivity in areas outside the center of the image. In both of the examples above, it would be easy for this type of exposure metering to be misled and give the incorrect exposure information. A more advanced AE technique is called spot metering. In this technique, the area that is being evaluated for exposure is much smaller than in the center weighted example. This allows the user to put the “spot” on the area deemed most important or interesting, and then to properly expose that area. An even more advanced version of this technique allows the user to specify multiple spots on the image, and then allows the camera to determine the most optimal exposure from this information. This technique still has several pitfalls. First of all, it requires the user to be more involved in picture taking than many users prefer. In addition, when taking pictures very quickly (perhaps during a sporting event) the user may simply not have the time to put the “spot” on the critical point(s) in the image. Today, high-end SLR cameras use a multiple zone evaluative system to arrive at the correct exposure information. For example, the Nikon D100 digital SLR uses a 10-zone matrix system to arrive at its exposure. This type of multi-zone system, when combined with the algorithms derived from analyzing exposure on thousands of images by camera manufactures, allows for a very fast and generally very accurate AE system. As we shall see, the SiP-1280 uses the same advanced scheme which provides for very fast, flexible, and accurate AE data. In addition, the SiP-1280 allows its features to be combined in interesting ways to provide even more functions for the camera user. © 2004 NuCORE Technology Inc Page 10

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The SiP-1280 provides an exceptionally flexible means to provide AE functionality. Somewhat similar in approach to the AF technique, the incoming image is divided into a series of blocks laid out in a grid over the image to create the number of zones and zone pattern desired by the OEM. Both the size and number of the blocks is configurable under software control. The minimum number of rows and columns is eight each for a total of 64 blocks up to a maximum of 32 rows and 32 columns for a total of 1024 blocks. The figure below gives an idea on how these blocks are arranged:

AE BLOCK

AE BLOCK 0

AE BLOCK 1

AE BLOCK 2

AE BLOCK 30

AE BLOCK 31

AE BLOCK 32

AE BLOCK 33

AE BLOCK 34

AE BLOCK 62

AE BLOCK 63

AE BLOCK 64

AE BLOCK 65

AE BLOCK 66

AE BLOCK 94

AE BLOCK 95

AE BLOCK 992

AE BLOCK 993

AE BLOCK 994

AE BLOCK 1022

AE BLOCK 1023

Variable Size

Variable Size

AE Whole Screen

Variable Size

Variable Size

Whole Image Size

Each AE block has several modes that may be used to measure the incoming light. In the simplest mode, each block simply accumulates all of the pixel values. In more advanced modes, the AE blocks can also keep track of the lightest and darkest pixels and optionally ignore or saturate light levels outside of a preset range. This gives the camera manufacturer unparalleled flexibility in arranging the AE data to provide the very best in auto exposure control. Besides being very flexible and powerful in AE configuration, the SiP-1280 is very fast at processing AE data due to a dedicated hardware engine that quickly analyzes AE data for each frame of information. Like the AF data, the AE information is also moved to main memory via Direct Memory Access (DMA) to minimize the loading on the embedded CPU while processing images. Thus, in draft or preview mode the AE circuits can fully analyze the AE data at 30 fps and seamlessly provide this information to the embedded CPU for use in determining exposure control.

SiP-1280 Differentiator →

The SiP-1280’s hardware based AE circuitry can support up to 1024 blocks. The no compromise circuitry is fast and flexible to arrive at optimal exposures picture after picture.

© 2004 NuCORE Technology Inc Page 11

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

While we may not realize it, the human eye is a marvelous instrument and it does many things for us automatically without us having to do anything, or even notice that it is doing anything. For example, when measured with instruments, a room lit only with incandescent bulbs will appear to have a very strong orange yellow cast to the light. On the other hand, a room lit with only fluorescent lights will have a very strong greenish cast when measured with those same instruments. Yet our eyes and brain will detect the light shift and compensate for it without us even realizing that compensation has been done. The figure below shows several potential light situations and exaggerated effects on the matching RGB light levels. 100%

B

G

Micro Lens

R

Color Filter Photo Diode

100%

G

R

G

R

B

Light

Image Sensor

Transparency

100%

B Wave Length

When faced with differing light situations, a digital camera must perform a similar compensation to what the eye does. This process is called Auto White Balance (AWB), and it is the means by which the light levels inside a camera are balanced to match the incoming light. As the picture below shows, the AWB process in a camera is more complex due to the many things that can affect color balance independent of the image lighting.

© 2004 NuCORE Technology Inc Page 12

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance Original Scene Color Information

Actual Scene

Camera Lens Assembly

Shift in Color Information - Lens

MicroLens Shift in Color Information – CCD (micro lens)

Color Gain Adj. NDX-1260

Color Filter CCD Cells CCD Sensor

NDX-1260

Looking at the CleanCapture NDX-1260, NuCORE believes that it is essential that the AWB process must first be done in the analog domain by the Analog Front End. Failure to do so will result in unacceptable color artifacts, which will drastically reduce print quality and degrade image compression results. However, it may at times be desirable to perform AWB in both the digital and analog domains, and the SiP-1280 provides the capability to perform AWB in the digital domain. In this case, the SiP-1280 will seamlessly apply the AWB offset into the incoming image data at full speed, without any performance penalty. Now let’s tie together AWB, AF, and AE to see how the camera user can get usable and advanced features. As an example, the camera manufacturer can combine the AF and AE data together to arrive at the optimal solution. To use the earlier example of a family member sitting in a backlit window, the camera can note that there is a large area on the image that is underexposed, and in addition the camera can determine that the underexposed area is closer to the camera than the rest of the image. This could allow the camera to correctly determine that the person in the window should be exposed correctly, and the rest of the image should be allowed to be overexposed. In addition, the camera could detect the color balance of the person and do AWB based on the interior lighting and not the exterior lighting.

© 2004 NuCORE Technology Inc Page 13

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance But let’s go several steps further in search of our superior image capture. As will be discussed later, the SiP-1280 has a skin tone detect feature. And perhaps the SiP-1280 has been put in a portrait mode, so the camera knows that you are taking pictures of people. This allows the SiP-1280 to use skin detection to locate the person in the image, to then use AE and AF to arrive at the optimal values for that person, to use Y Gamma to create a good range of light tones in the image, and to finally adjust the image processing filters for the incoming image such that the person has a slightly softened image. SiP-1280 Differentiator →

Only the SiP-1280 provides very flexible AE, AWB, and AF capabilities which enable the camera manufacturer to provide uncompromising performance and superior results.

Color Correction The color correction block is used to compensate for cross color bleeding in the color used with the image sensor (see Bayer to RGB/CYMG to RGB Block for more details). The block uses a set of matrix coefficients (a11 to a33) in the matrix calculation shown below, to do the correction:  R ′   a11 a12 a13   R         G ′  =  a 21 a 22 a 23  ×  G      B′   a    31 a 32 a 33   B  In the development of a new camera, after the integration of the key camera components (ex. lens, image sensor, and image processor), these coefficients are tweaked by the OEM in the final tuning stage for optimal image quality. To allow the OEM to quickly do this tuning, NuCORE provides a CCTT (CleanCapture Tuning Tool) application for real-time customization and tuning of the imaging pipeline. Below is a screen capture of the intuitive user interface for tuning the color correction:

The SiP-1280 PDK (Product Development Kit) comes with the CCTT SiP-1280 Differentiator → application for real-time tuning of color correction. © 2004 NuCORE Technology Inc Page 14

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

Edge Enhancement Edge enhancement is used to sharpen edge detail and enhance the visibility of a boundary between light and dark tones in an image. It is performed by a mathematical formula that is applied across the image. Most digital cameras will, as a part of the normal image processing, apply some level of edge enhancement to counteract the softening effects from the low-pass optical filter and interpolation of colors during the Bayer RGB or complementary conversion to RGB process (see Bayer to RGB/CMYG to RGB Block for more details). The problem with typical edge enhancement algorithms is that it increases the visibility of “jaggies” or rough edges, and decreases the overall SNR (Signal to Noise Ratio) of the image. A key part of the CleanCapture technology is the proprietary edge enhancement algorithm used by the SiP-1280, which can produce a sharp natural image without introducing any noise or other unwanted secondary effects. Below is a side-by-side image comparison of conventional edge enhancement on the left and SiP-1280’s sharper and cleaner edge enhancement on the right. For those readers who print this document, please note that the differences in the images are best seen on photo quality paper: Conventional Edge Enhancement

SiP-1280 Edge Enhancement

As with all the other parts of the SiP-1280 image pipeline, the edge enhancement is designed to be customizable by the OEM. Using the CCTT (CleanCapture Tuning Tool) application for real-time customization and tuning, the parameters of the edge © 2004 NuCORE Technology Inc Page 15

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance enhancement algorithm can be adjusted to get the desired level of sharpness. Below is a screen capture of the intuitive user interface for tuning the edge enhancement:

A key part of the CleanCapture technology is the proprietary edge enhancement algorithm used by the SiP-1280, which can produce a SiP-1280 Differentiator → sharp natural image without introducing any noise or other unwanted secondary effects.

Sensitivity

RGB Gamma Correction The human eye does not detect light intensity in a linear fashion. This means that humans can detect greater color changes in darker luminance areas of an image then they can in lighter luminance areas. As seen in the graph below the relationship is logarithmic in nature:

Luminance In digital imaging, RGB gamma correction uses this phenomenon to enhance the picture where the dynamic color range is logarithmically shifted to the darker luminance area. © 2004 NuCORE Technology Inc Page 16

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance This gamma correction is applied when the image is in the RGB color space (while the Y-gamma compensation, to be introduced later, is applied when the image is in the YUV color space). Unlike other typical solutions that use a linear approximation as a shortcut for the logarithmic calculation, the CleanCapture SiP-1280 SiP-1280 doesn’t compromise and uses a true logarithmic algorithm for RGB Differentiator → gamma correction. In addition, it can be easily customized for camera tuning, even for every captured image, in real time. Chroma Suppression Chroma suppression is used to improve the look and feel of the color SNR by reducing false colors from the AFE and the interpolation process in the Bayer to RGB conversion. Since the NDX-1260 doesn’t introduce any false color, if it is used in conjunction with the SiP-1280, very little chroma suppression is required. The SiP-1280 uses a proprietary adaptive approach that allows the algorithm to change the degree of chroma suppression depending on SiP-1280 Differentiator → the characteristics of a particular image. Most other solutions only allow for uniform chroma suppression over the entire image. Y Gamma Compensation Ansel Adams first proposed the Zone system as a way of allowing photographers to reliably capture the image that they visualized onto film. As part of this system, the photographer adjusts his or her film exposure, film processing, print contrast, and print processing to modify such things as how the low, mid, and high light levels are realized on a print. This may seem overly complicated, but it leads to an exceptionally powerful tool for adjusting picture quality. And just as Ansel Adams was able to do with film, the digital camera user can make similar adjustments to his or her images as well. Let’s look at a few example images to show the basic concepts involving Y gamma. The “Y” in Y gamma is shorthand for luminance, or the relative brightness of something. This gamma is applied when the image is in the YUV color space, as opposed to RGB gamma, which is applied in the RGB color space. To help illustrate luminance by itself and avoid complications caused by color information, the initial images will be black and white.

© 2004 NuCORE Technology Inc Page 17

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The above image is a black and white converted image of a test target image. This image is the base used for comparison in later images. The image below has had its gamma adjusted such that mid gray tones have been moved more white. Note however, that this is not a simple brightness adjustment. The black text on the wine bottle is still black, which would not be the case if we had just uniformly adjusted the brightness.

The next image shows the effect of adjusting gamma such that the mid tones are moved towards black. Once again, this is not just a simple brightness adjustment, as the top of the box on the left is still white.

The previous examples have shown how we can move the mid tones of an image towards white or black without affecting the pure whites or blacks in the image. This is the real advantage of adjusting Y gamma: the mid tones of an image can be moved lighter or © 2004 NuCORE Technology Inc Page 18

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance darker without impacting the near white or near black tones. Now let’s add color and see how adjusting Y gamma can really improve an image.

Above is the original image before it was converted to black and white. While this is an acceptable image, the image looks very “flat” and is not very appealing. The below image is with modified gamma. Note how much more alive the image looks. However, because we adjusted Y gamma and not brightness, the blacks are still black and the whites are still white.

The SiP-1280 has a hardware based capability to adjust the Y-gamma value either during image capture or alternately during a post-processing. As this capability is hardware based, there is no trade-off between quality and time as there might be in a DSP-based design. Additionally, NuCORE’s no-compromise approach to image processing pays appropriate attention to such things as the number of bits used in gamma calculations. This capability, when combined with the SiP-1280’s advanced auto exposure (AE) © 2004 NuCORE Technology Inc Page 19

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance capabilities, allows the SiP-1280 to compress or expand the mid tones of an image to arrive at an optimal gamma for the image without quality trade-offs.

SiP-1280 Differentiator →

Unlike DSP based solutions, which don’t have the processing performance and therefore the time to do a quality job – or only do RGB gamma correction, the SiP-1280 can do high quality precision Y gamma in real time to deliver superior image quality.

SiP-1280 Differentiator →

Y-gamma also makes the SiP-1280 a perfect choice for Hybrid type devices, which require both video and still image processing. Ygamma is applied in the YUV color spac,e which is the same color space used to process video.

Skin Tone Detection The SiP-1280 has a unique DSC feature, skin tone detection, which is used to locate human skin (primarily faces) in an image and apply different digital filters. Usually the edge enhancement is reduced to faces to give a more a natural look, but the hardware is flexible enough to allow other filtering techniques. Alternately, this logic can also be used as a chroma-key for doing digital overlay special effects. As with the other sections of the image pipeline, the sophisticated algorithms used in skin tone detection create a smooth edge along the face and significantly minimize misdetection in other parts of the image. In industrial markets this feature can be used to perform face recognition in security cameras. The images below show the original image on the left, and the accuracy and smoothness of NuCORE’s skin tone detection algorithm on the right:

© 2004 NuCORE Technology Inc Page 20

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance As with all the other parts of the SiP-1280 image pipeline, skin tone detection is designed to be customizable by the OEM. Using the CCTT (CleanCapture Tuning Tool), the parameters of the skin tone detection algorithm can be localized for different skin tones. Below is a screen capture of the intuitive user interface for tuning the skin tone detection:

The SiP-1280 has the ability to do sophisticated skin tone detection SiP-1280 which allows special processing for a more natural look for faces Differentiator → without introducing artifacts into other areas of the picture Down Sampling Filter The down sampling filter is used to reduce the image size to D1 resolution (720x480) for output to the SiP-1280’s digital video port. This digital video can be used to connect to a DV CODEC for miniDV DVCs, or to an MPEG2 or MPEG4 CODEC in a Hybrid type device. The SiP-1280 has integrated support for video peripherals, a key SiP-1280 Differentiator → requirement for Hybrid type devices Digital Effects The SiP-1280 can perform the following digital effects: 1. Black and white – Converts the image from color to black and white. 2. Sepia - Sepia toning is used to give black-and-white and color photographs a warmer or antique appearance. 3. Negative - Inverts the colors in an image. This makes a positive black-and-white image negative or a positive from a black-and-white negative. 4. Image Mirroring – Flips the image in either the horizontal or vertical direction. RGB to YCrCb Block, YCrCb to RGB Block, and the YCrCb Conversion Block © 2004 NuCORE Technology Inc Page 21

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance The SiP-1280 can convert RGB to YCrCb for the high-performance JPEG engine. YCrCb (or sometimes called YUV) is the color scheme required for JPEG and MotionJPEG compression. Y is the luminance (brightness) component and CrCb are the color components. The YCrCb data is then passed through the YCrCb Conversion from a 4:4:4 sampling to a 4:2:2 sampling. The 4:2:2 means there is now twice as much data for the luminance as there is for color, which is done because the human eye is better at detecting contrast and intensity than color changes. Back-end Resizer The back-end resizer is responsible, if needed, for downsizing to the desired final image size. As an example, the user of a 5 megapixel camera may want to lower the resolution to 1 megapixel so that the file size is small enough to send across the internet. The starting resolution of 2560 x 1920 needs to be converted to 1280 x 960, where the first number is the resolution in the horizontal or “x” direction and the 2nd number is resolution in the vertical or “y” direction. The back-end resizer is also required in Hybrid applications where the high-resolution image sensor data is scaled down to NTSC or PAL video resolutions. While other solutions have limitations on sizes, the SiP-1280 has a very flexible back-end resize that can downsize from any image size to any other image size, as shown in the diagram below:

Y

Such that X`< X Y`< Y

Y`

X

X`

SiP-1280 The SiP-1280 has a very flexible high-quality back-end resizer. Differentiator →

Digital Zoom Engine Digital zooming is the process by which an image is scaled from one size to another. Generally speaking, this technique is used to enlarge the size of an image to provide an apparent increase in lens focal length. Zooming is a complex and subtle process – you are essentially creating pixels from where there were none before – so the interpolation technique that you use to zoom can be critical. Zoom quality is especially important in image zooming for video applications, where a large amount of zooming is generally more tolerable to the user than in still applications. For most digital cameras, the digital zoom method used is bilinear. In this type of zoom, a simple average of the four nearest pixels to the generated pixel is used to determine its © 2004 NuCORE Technology Inc Page 22

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance value. This method is relatively inexpensive in chip area and performance costs, but it can also lead to blurred edges and distortion artifacts in the zoomed images. The only exception to this general rule is in geometric types of pictures, in which case bi-linear interpolation can arrive at slightly superior results. The diagram below shows the basic concepts behind bilinear interpolation. As can be seen, only four adjacent pixels must be sampled to determine the value for the fifth (or interpolated) pixel. Thus the logic required for this type of interpolation is relatively simple.

Bi-cubic interpolation is a more advanced technique that gives more accurate edge detail and less pixelation than bi-linear interpolation. Bi-cubic interpolation uses sixteen pixels in a 4x4 matrix to generate the interpolated pixel. This method has superior results, however it is more complex to implement due to the much larger number of calculations that must be performed to generate each pixel. The pictures below demonstrate the technique used for bi-cubic interpolation.

© 2004 NuCORE Technology Inc Page 23

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance

As can be seen from the above picture, in bi-cubic interpolation each new pixel is created from the nearest sixteen pixels. The combination of more pixels being sampled with the more advanced bi-cubic algorithm, results in a much more optimal value for the interpolated pixel. The SiP-1280 can perform either bilinear or bi-cubic interpolation. In addition, since this interpolation is hardware based, there is no performance trade-off required when the more advanced bi-cubic technique is preferred. SiP-1280 Differentiator →

Unlike other typical solutions, which use a bilinear approximation as a short cut for the interpolation, the SiP-1280 doesn’t compromise and uses an advanced bi-cubic interpolation technique for superior image quality.

JPEG and MotionJPEG Engine Typical DSCs use JPEG to compress images from 3:1 to 6:1, usually with little or no image degradation. The JPEG process divides the image into 8 by 8 pixel blocks and then calculates the discrete cosine transform (DCT) of each block. A “quantizer” rounds off the DCT coefficients according to the quantization matrix. This step produces the "lossy" nature of JPEG, but allows for large compression ratios. JPEG's compression technique © 2004 NuCORE Technology Inc Page 24

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance uses a variable length code on these coefficients and then writes the compressed data stream to memory. For decompression, JPEG recovers the quantized DCT coefficients from the compressed data stream, takes the inverse transforms, and displays the image. Since NuCORE’s CleanCapture technology significantly reduces color noise and artifacts, the SiP-1280’s JPEG engine can achieve close to double the compression (or in other words half the file size) of other solutions with the same image quality. Here is a good analogy - when computer graphics (ex. animation) are compressed, you get much higher compression ratios as compared to a captured image with same resolution. This happens because the objects inside the computer graphics have very uniform colors, which can easily be compressed having zero noise or artifacts. Bottom line - less noise, higher compression ratios! SiP-1280 Differentiator →

NuCORE CleanCapture Technology allows the SiP-1280 to compress images, which have almost half the file size and the same quality. This means the user can store almost twice as many images on their flash memory.

MotionJPEG applies the same JPEG compression, at a much faster rate, to each individual frame of a movie. The SiP-1280 has a very fast JPEG engine that run at the 30 frames/second at VGA resolution video and 15 frames/second at HDTV resolution (1280x720). SiP-1280 Differentiator →

The SiP-1280 JPEG engine is fast enough to compress at HDTV resolution.

AFE interface and Front-end Resizer The AFE interface is used to connect to an Analog Front End device and connects seamlessly to the NDX-1260. The Front-End resizer is used for downscaling the image for viewing/previewing mode. This mode allows for power saving during real-time previewing when combined with the CCD sampling output mode. The SiP-1280 has a fast 81MHz 12-bit AFE interface. In addition, it SiP-1280 supports RGB interleaved data stream for 3CCD type systems, Differentiator → typically used in professional grade DVCs. OSD Engine The OSD (On-Screen Display) is responsible for displaying user information and camera menu options to the LCD screen on the camera, which can be overlaid onto the preview © 2004 NuCORE Technology Inc Page 25

The CleanCapture™ SiP-1280 Image Processor A No-Compromise Approach to Image Quality and Performance of an image. The OSD engine can also be used to superimpose time and date stamps (as well as any other text) on the final capture image. Some OSD engines are only character-based and support a single SiP-1280 overlay, but NuCORE has designed a no-compromise OSD engine Differentiator → that uses a full color (24-bit) bitmap that supports 2 overlay planes. ARM9™ Embedded Processor Part of the SOC (System-On-a-Chip) concept is an embedded high-performance 32-bit ARM9™ processor, which has an 8K instruction cache, 8K data cache, and a MMU which supports virtual addressing. The processor runs a real-time operating system and file system manager that controls and coordinates all aspects of the camera’s functions. In conjunction with the embedded processor, NuCORE provides a platform solution with a suite of developer tools. This includes the NuCamera Firmware, a comprehensive API, which provides a hardware abstraction layer allowing engineers to focus on functionality and programming down to the register level. This dramatically reduces time-to-market for new camera development. The embedded processor also runs AF, AE, and AWB algorithms developed by the camera manufacturer (see AF-AE-AWB Statistics). Digital I/O As stated earlier, the SiP-1280 is architected to be a SOC and can be seamlessly connected to variety of peripherals including, digital LCD panels, USB 2.0 (High Speed) devices, Compact Flash memory, SD memory, SDIO Wireless LAN Modules, Multimedia Card memory, ATA hard drives, optical drives, AC-97 compliant audio codecs and a digital video port for external MPEG2 and MPEG4 encoding/codecs. Analog I/O The SiP-1280 has 5 internal high precision DACs (Digital Analog Converters) which are used for interfacing to an embedded analog LCD and/or an external TV. As was talked about earlier, not only can the SiP-1280 directly connect to a standard TV via composite and S-video, but also has DirectView™/HD which allows the direct connection to a HDTV monitor via SMPTE compliant component video (YCrCb):

Back of HDTV

© 2004 NuCORE Technology Inc Page 26