Color interpolation with variable color ratio considering cross-channel correlation Sang Wook Park Moon Gi Kang, MEMBER SPIE Yonsei University Department of Electrical and Electronic Engineering 134 Shinchon, Seodaemun-Ku Seoul, Korea E-mail: [email protected]

Abstract. A color interpolation algorithm is necessary to acquire color images of good quality with a digital still camera in which a single CCD image sensor is equiped. Since a color filter array is adopted, we must interpolate missing color values in the red, green, and blue channels at each pixel location. Inappropriate color interpolation causes color artifacts, resulting in degraded color resolution. A color interpolation algorithm with a variable color ratio is proposed by incorporating crosschannel correlation. The color ratio of weighted sums in the color channels is utilized instead of the weighted sum of color ratios, and the color ratio varies with respect to the cross-channel correlation at each pixel. The proposed algorithm is composed of initial color interpolation step and the updating step of the green channel, where the updating step is carried out iteratively. Furthermore, a minus weighted sum based on the external divide, a modified edge indicator function, and a 5⫻5 support region for the red and blue channels are also adopted to improve the performance. With the proposed algorithm, we have achieved improvements in the image quality especially along the edges and in the detail of the image by removing color artifacts. The superiority of the proposed algorithm is demonstrated in the experimental results. © 2004 Society of Photo-Optical Instrumentation Engineers. [DOI: 10.1117/1.1631000]

Subject terms: charge-coupled device; color filter array; color interpolation; the Bayer pattern; cross-channel correlation. Paper 030184 received Apr. 21, 2003; revised manuscript received Jul. 14, 2003; accepted for publication Jul. 15, 2003.

1

Introduction

Most digital still cameras adopt single CCD image sensor commonly. However, since the single CCD image sensor can perceive only the brightness of a pixel, a color filter array 共CFA兲 must be used to obtain color information at each pixel location. The Bayer pattern1 shown in Fig. 1 is widely used as a CFA. Since each pixel captures only one color component, other two colors must be interpolated from the neighboring pixel values. If color interpolation is carried out inappropriately, the image suffers from various color artifacts, resulting in degraded color resolution. Therefore an efficient color interpolation algorithm is required to overcome this physical limitation of the CCD image sensor and to increase the resolution of color images. Several color interpolation algorithms for a CCD image sensor were proposed since the late 1980s. The algorithms were proposed based on the assumption that the hue changes smoothly in the neighborhood.2,3 Cok used bilinear interpolation for the luminance channel, and the chrominance channels were interpolated from the red or blue hue, which is the ratio of the chrominance channel value over the green channel value.2 Adams interpolated the logarithm of the hue.3 Color interpolation algorithms using a complementary color filter array were developed where the local color signal correlation was utilized to reduce color moires.4,5 Color correlation interpolation was performed by using 2-D low-frequency components.6,7 Meanwhile, interchannel correlation was considered in Refs. 8 –12. These 34

Opt. Eng. 43(1) 34–43 (January 2004)

methods are called edge-directed interpolation. Hibbard calculated the first order horizontal and vertical gradients at a missing green pixel which is luminanace channel pixel.8 Laroche and Prescott utilized the chrominance channels to take the gradients, and interpolating direction was determined by the gradients.9 Hamilton and Adams used secondorder derivatives of the chrominance channels for the correction of the green channel.12 Zen et al. interpolated the green pixel values by averaging the two median values of the four horizontally and vertically neighboring green pixel values,13 and Kim et al. exploited the median value of the three neighboring green pixel values for simple hardware implementations.14 The discriminated color correlation based on horizontal and vertical gradients was used in the color interpolation.15 Although these conventional algorithms were easy to be implemented in hardware, they could not remove color artifacts completely. To reduce the color artifacts and acquire the image of improved color resolution, other algorithms have been proposed. Edge-adaptive color interpolation algorithms were proposed in Refs. 16 and 17. Kimmel used both smooth hue transition and edge-directed interpolation for demosaicing. An edge indicator function was formed of the reciprocal of the first-order derivative in the green and chrominance channels. Then color ratios were multiplied by the weights calculated by the edge indicator function. Missing pixel values were interpolated by multiplying the interpolated ratios by a channel value. Each channel was corrected iteratively according to the color ratio rule.16 Hur and Kang17

0091-3286/2004/$15.00

© 2004 Society of Photo-Optical Instrumentation Engineers

Park and Kang: Color interpolation with variable color ratio . . .

Fig. 1 RGB color filter array (the Bayer pattern).

proposed a new edge indicator function that used an exponential function. He eliminated false color artifacts that resulted from the color image formation model in Ref. 16 by adopting a switching algorithm based on the color edge detection.17 Chang et al. interpolated missing channel pixels using a threshold to choose gradients. Gradients were computed in the 5⫻5 support region and interpolation was carried out by the pixels corresponding to the gradients that were higher than the threshold.18 Cok applied pattern recognition in color interpolation.19 Adams20 and Glotzbach et al.21 explored sampling theory for the color interpolation problem. Color interpolation was performed by restorationbased methods.22–24 Mukherjee et al. exploited simulated annealing for the restoration process.24 A color interpolation method based on wavelet transform was proposed by Gunturk et al., and the projection of subbands on a convex set was conducted to reconstruct missing channel pixels.25 Although color artifacts have been reduced by utilizing the mentioned complicated algorithms, color artifacts still remained, especially along the edges and in the highfrequency detail of the image, and the image does not provide adequate color resolution. To remove the color artifacts that remain in the image, and to obtain the image of definite color resolution, color interpolation with a variable color ratio considering cross-channel correlation is proposed in this paper. This paper is organized as follows. Section 2 presents the assumptions and theoretical problems of conventional algorithms. A detailed explanation for theoretical improvements including a variable color ratio and considering cross-channel correlation and the proposed algorithm are presented in Sec. 3. Section 4 present its experimental performance and comparisons with other algorithms, and we conclude with Sec. 5. 2

Assumptions and Theoretical Problems of Conventional Algorithms In the color ratio rule, which can be derived from the photometric image formation model,26 it is assumed that the color ratio is constant in an object. The surface of an object reflects all incident light equally in all directions in an ideal Lambertian surface, and a Mondrian is an object made of patches with constant reflectivity. A Lambertian surface of a Mondrian enables a color image model to be simplified. Each color channel is composed of the albedo multiplied by

the projection of the surface normal onto the light source direction in this model. The albedo is the fraction of the light reflected by the surface, and it is different for each color channel. If an object is considered as a Lambertian surface of a Mondrian, the albedos are the same in the object.16 However, the color ratio rule based on the photometric image formation model is an oversimplified assumption for the image, the color ratio is not constant, and it varies sharply along the edges. To consider this variation of the color ratio, the edge information was adopted in the edge adaptive color interpolation algorithms.16,17 The color ratio was multiplied by the weight computed by an edge indicator function in the color channels. Although the edge indicator function could reflect edge information, it was insufficient to show cross-correlation between the color channels. Therefore new weights that can reflect crosschannel correlation are required. In the proposed algorithm, the color ratios are multiplied by the weights that vary according to the cross-channel correlation at each pixel, and the initially interpolated value of the green channel is adjusted to the color ratio rule by incorporating the updated values of the green channel into the initially interpolated value. Thus the color ratio rule, which can not hold along the edges, can be applied to the updating process of the green channel without causing color artifacts. 3

Color Interpolation with Variable Color Ratio Considering Cross-Channel Correlation

3.1 Variable Color Ratio in the Green Channel The color ratio of weighted sums in color channels is utilized in this paper, since the color ratio at one pixel differs more largely from the color ratios at the neighboring pixels than the color ratio of several pixels’ averages in color channels. If the color ratio changes less sharply, the color ratio rule, which assumes that the color ratio is constant or changes smoothly, can be appropriately adopted along the edges and in the high frequency detail of the image. The updated green values at location R in Fig. 1, which are determined by two color ratios, are given as follows: G i,R j ⫽R i, j

G i,B j ⫽B i, j

R R w i⫺1,j G i⫺1,j ⫹w i,R j⫺1 G i, j⫺1 ⫹w i⫹1,j G i⫹1,j ⫹w i,R j⫹1 G i, j⫹1 R R w i⫺1,j R i⫺1,j ⫹w i,R j⫺1 R i, j⫺1 ⫹w i⫹1,j R i⫹1,j ⫹w i,R j⫹1 R i, j⫹1

B B w i⫺1,j G i⫺1,j ⫹w i,B j⫺1 G i, j⫺1 ⫹w i⫹1,j G i⫹1,j ⫹w i,B j⫹1 G i, j⫹1 B B w i⫺1,j B i⫺1,j ⫹w i,B j⫺1 B i, j⫺1 ⫹w i⫹1,j B i⫹1,j ⫹w i,B j⫹1 B i, j⫹1

,

共1兲

,

共2兲

A where w i⫹n, j⫹m is obtained as A w i⫹n, j⫹m ⫽

1 , exp共 C1 n,m / ␴ 兲 ⫹exp共 C2 n,m / ␴ G 兲 ⫺1 G

共3兲

where ␴ G is the standard deviation of the green channel, C1 n,m and C2 n,m represent the absolute differences for each directional derivative of the green channel and a channel at (i, j), respectively, which are given as follows: C1 n,m ⫽2 兩 G i⫹n, j⫹m ⫺G i⫺n, j⫺m 兩 , C2 n,m ⫽2 兩 A i⫹2n, j⫹2m ⫺A i, j 兩 , Optical Engineering, Vol. 43 No. 1, January 2004

共4兲

35

Park and Kang: Color interpolation with variable color ratio . . .

where A becomes either the red or blue channel and (n,m)⫽(⫺1,0), 共0,⫺1兲, 共⫹1,0兲, 共0,⫹1兲. The updated value of the green channel can be evaluated as G ⬘i, j ⫽ 共 1⫺b 兲 G i, j ⫹b 关 aG Bi, j ⫹ 共 1⫺a 兲 G Ri, j 兴 ,

共5兲

where G i, j is the initially interpolated value of the green channel, and the explanation for the interpolation of the green channel is presented later. Note that G ⬘i, j is an updated value of the green channel, and a and b are weights. The weight a is expressed as follows:

a⫽

再

1

if B i, j ⬎G i, j ⫹T,B i, j ⬎R i, j ⫹T, 兩 G i, j ⫺G i,R j 兩 ⬎T

0

if G i,R j /R i, j ⬍1

G i,R j /R i, j ⫺1

otherwise,

共6兲

where 0⬍a⬍1. The B i, j ⬎G i, j ⫹T and B i, j ⬎R i, j ⫹T are conditions such that even though the originally acquired color at the pixel was red thus the color ratio of the green over the red must be used for the updating process, the color ratio of the green over the blue should be exploited for the updating, if the blue color has the highest value of all the channels, and the pixel’s color is blue. The threshold T is added to the inequalities to recognize the color of the pixel as a bluer color. The 兩 G i, j ⫺G Ri, j 兩 ⬎T is another condition such that if the difference between G i, j and G Ri, j is large, the ratio of the green over the red is greatly large or small compared with the ratio of the green over the blue. Therefore the color ratio of the green over the blue will be appropriate to update the initially interpolated value. When the updating process of the green channel is performed at location B in Fig. 1, the threshold is set as T⫹40. The threshold for recognizing the color of the pixel as a redder color is larger than as a bluer color, since the red channel image is more similar to the green channel image than the blue channel image. The G Ri, j /R i, j ⬍1 is the condition that if the weighted sum in red channel is larger than in green channel, thus the pixel’s color is red, then the color ratio of green over red should be used. If none of the two chrominance values is large, the two updated green values by both the ratios are mixed together. The b is acquired as follows:

b⫽

再

R i, j /G i, j k ⫺1 wR/G i, j k

wR/G i, j ⫺1 R i, j /G i, j

if R i, j /G i, j ⬎wR/G i, j 共7兲 otherwise,

冉

R i⫺1,j R i, j⫺1 R i⫹1,j R R wR/G i, j ⫽ w i⫺1,j ⫹w i,R j⫺1 ⫹w i⫹1,j G i⫺1,j G i, j⫺1 G i⫹1,j R i, j⫹1 G i, j⫹1

冊冒

The missing pixels in the red and blue channels are interpolated by using the color ratio rule from the beginning, for those chrominance channels do not have enough original pixels. However the interpolation of a single channel will be more appropriate, if the missing color at a pixel is larger than the other colors around the neighboring pixels, thus the color at the pixel is the color of the missing channel. The interpolation of the red channel at location B in Fig. 1 and Fig. 2共a兲 is given as R i, j ⫽ 共 1⫺t 兲 R G i, j ⫹t 共 e i⫺1,j⫺1 R i⫺1,j⫺1 ⫹e i⫺1,j⫹1 R i⫺1,j⫹1 ⫹e i⫹1,j⫺1 R i⫹1,j⫺1 ⫹e i⫹1,j⫹1 R i⫹1,j⫹1 兲 / 共 e i⫺1,j⫺1 ⫹e i⫺1,j⫹1 ⫹e i⫹1,j⫺1 ⫹e i⫹1,j⫹1 兲 ,

共9兲

where e i, j is a modified edge indicator function defined as e i⫹n, j⫹m

⫽

1 exp共 D1 n,m / ␴ G 兲 ⫹0.55 exp共 D2 n,m / ␴ G 兲 ⫹0.55 exp共 D3 n,m / ␴ G 兲 ⫺2

,

Optical Engineering, Vol. 43 No. 1, January 2004

where D1 n,m , D2 n,m , and D3 n,m are the absolute differences for each directional derivative of the green channel at (i, j), respectively, and ␴ G is the standard deviation of the green channel. At location B in Fig. 1 and location R in Fig. 3共a兲, D1 n,m ⫽ 兩 G i⫹n, j⫹m ⫺G i, j 兩 , D2 n,m ⫽ 兩 G i⫹2n, j⫹m ⫺G i⫹n, j 兩 ,

共11兲

D3 n,m ⫽ 兩 G i⫹n, j⫹2m ⫺G i, j⫹m 兩 ,

R R ⫹w i,R j⫺1 ⫹w i⫹1,j 共 w i⫺1,j

⫹w i,R j⫹1 兲 . 36

3.2 Variable Color Ratio in the Red and Blue Channels

共10兲

where 0⬍b⬍1, k is a constant that is positive, and wR/G i, j is the weighted sum of the color ratios of the red channel over the green channel described as

⫹w i,B j⫹1

The weight b is the ratio of the two color ratios, one of which is the ratio at 1 pixel and the other of which is the weighted sum of the ratios at the neighboring pixels. It is calculated by determining the ratio of the larger one over the smaller one. If the color ratio at the updated pixel is greatly close to the ratios at the neighboring pixels, b has a value of about 0. In the opposite case, b has a value of about 1. Although the initially interpolated value of the green channel is less appropriate than the updated value of the green channel according to the color ratio rule, the initial green value should not be totally exchanged by the updated green value, if the color ratios do not change largely, and the color ratio rule is already satisfied. The constant k is a scaling factor for b. If k is small, a small fraction of the updated value of the green channel is mixed with the initially interpolated value of the green channel at an iteration, thus the updating is slow. If k is large, the updating is quick.

共8兲

where (n,m)⫽(⫺1,⫺1), 共⫺1,⫹1兲, 共⫹1,⫹1兲, 共⫹1,⫺1兲. At location G1 in Figs. 2共b兲 and 3共b兲 and location G2 in Figs. 2共c兲 and 3共c兲, D1 n,m , D2 n,m , and D3 n,m are given as follows:

Park and Kang: Color interpolation with variable color ratio . . .

D1 n,m ⫽ 兩 G i⫹n, j⫹m ⫺G i, j 兩 ,

The

D2 n,m ⫽ 兩 G i⫹n⫺k, j⫹m⫺l ⫺G i⫺k, j⫺l 兩 ,

共12兲 共 R i⫺1,j⫺1 ⫹R i⫺1,j⫹1 ⫹R i⫹1,j⫺1 ⫹R i⫹1,j⫹1 兲 /4

D3 n,m ⫽ 兩 G i⫹n⫹k, j⫹m⫹l ⫺G i⫹k, j⫹l 兩 . where k⫽1; l⫽0, (n,m)⫽(⫺1,0), 共⫺1,⫺2兲, 共⫹1,⫺2兲, 共⫹1,0兲, 共⫹1,⫹2兲, 共⫺1,⫹2兲 at location G1; and k⫽0, l ⫽1, (n,m)⫽(⫺2,⫺1), 共0,⫺1兲, 共⫹2,⫺1兲, 共⫹2,⫹1兲, 共0, ⫹1兲, 共⫺2,⫹1兲 at location G2. Note R G i, j is computed as RG i, j ⫽G i, j 共 e i⫺1,j⫺1 R i⫺1,j⫺1 ⫹e i⫺1,j⫹1 R i⫺1,j⫹1 ⫹e i⫹1,j⫺1 R i⫹1,j⫺1 ⫹e i⫹1,j⫹1 R i⫹1,j⫹1 兲 / 共 e i⫺1,j⫺1 G i⫺1,j⫺1

⬎ 共 G i⫺1,j⫺1 ⫹G i⫺1,j⫹1 ⫹G i⫹1,j⫺1 ⫹G i⫹1,j⫹1 兲 /4

and (R i⫺1,j⫺1 ⫹R i⫺1,j⫹1 ⫹R i⫹1,j⫺1 ⫹R i⫹1,j⫹1 )/4⬎B i, j in Eq. 共14兲 are conditions such that if the red color at the pixel is larger than the other colors around the neighboring pixels, thus the color at the pixel is red, then the interpolated value of a single channel, which is

⫹e i⫺1,j⫹1 G i⫺1,j⫹1 ⫹e i⫹1,j⫺1 G i⫹1,j⫺1 ⫹e i⫹1,j⫹1 G i⫹1,j⫹1 兲 ,

共13兲

and t is calculated as

t⫽

¦

m i,R/G j

if

共 e i⫺1,j⫺1 R i⫺1,j⫺1 ⫹e i⫺1,j⫹1 R i⫺1,j⫹1 ⫹e i⫹1,j⫺1 R i⫹1,j⫺1

⫹e i⫹1,j⫹1 R i⫹1,j⫹1 兲 / 共 e i⫺1,j⫺1 ⫹e i⫺1,j⫹1 ⫹e i⫹1,j⫺1

R i⫺1,j⫺1 ⫹R i⫺1,j⫹1 ⫹R i⫹1,j⫺1 ⫹R i⫹1,j⫹1 4

⫹e i⫹1,j⫹1 兲

⬎B i, j ,

R i⫺1,j⫺1 ⫹R i⫺1,j⫹1 ⫹R i⫹1,j⫺1 ⫹R i⫹1,j⫹1 4 ⬎

G i⫺1,j⫺1 ⫹G i⫺1,j⫹1 ⫹G i⫹1,j⫺1 ⫹G i⫹1,j⫹1 4

⫺ ␴ i,R/G j

if

0

otherwise .

m R/G i, j ⫽

共14兲

␴ i,R/G j ⬎0.1

here, must be mixed with the interpolated value by using the color ratio rule which is R G i, j in this case. The situation R/G when t is ⫺ ␴ i, j or 0 is discussed in next subsection. The m R/G i, j is the local mean of log scale color ratios, and is computed as

log共 R i⫺1,j⫺1 /G i⫺1,j⫺1 兲 ⫹log共 R i⫺1,j⫹1 /G i⫺1,j⫹1 兲 ⫹log共 R i⫹1,j⫺1 /G i⫹1,j⫺1 兲 ⫹log共 R i⫹1,j⫹1 /G i⫹1,j⫹1 兲 , 4

where 0⬍m R/G i, j ⬍1. 3.3 Minus-Weighted Sum Based on the External Divide in the Red and Blue Channels The weight by the edge indicator function has a value from 0 to 1 in the conventional algorithms,16,17 since each weight

Fig. 2 Bayer pattern for the red channel: (a) location B, (b) location G1, and (c) location G2.

共15兲

is divided by the total sum of weights. The interpolated value is always within the maximum value of the neighboring pixels, and the minimum value of the neighboring pixels. However, the original pixel value is not often within the range of the maximum and the minimum of the neighboring pixels. The interpolated value in the red channel at location B is realigned from Eq. 共9兲, and is expressed as

Fig. 3 Bayer pattern for the blue channel: (a) location R, (b) location G1, and (c) location G2. Optical Engineering, Vol. 43 No. 1, January 2004

37

Park and Kang: Color interpolation with variable color ratio . . . G R i, j ⫽R G i, j ⫺t 关 e i⫺1,j⫺1 共 R i, j ⫺R i⫺1,j⫺1 兲 ⫹e i⫺1,j⫹1 G ⫻共 RG i, j ⫺R i⫺1,j⫹1 兲 ⫹e i⫹1,j⫺1 共 R i, j ⫺R i⫹1,j⫺1 兲

⫹e i⫹1,j⫹1 共 R G i, j ⫺R i⫹1,j⫹1 兲兴 / 共 e i⫺1,j⫺1 ⫹e i⫺1,j⫹1 ⫹e i⫹1,j⫺1 ⫹e i⫹1,j⫹1 兲 .

共16兲

Whereas the normalized weight divided by the total sum of the weights has a value from 0 to 1, (R G i, j ⫺R i⫺n, j⫺m ) can be any number that is positive or negative in Eq. 共16兲. The RG i, j is the average of the neighboring pixels of the red G channel. The 关 e i⫺1,j⫺1 (R G i, j ⫺R i⫺1,j⫺1 )⫹e i⫺1,j⫹1 (R i, j

␴ R/G i, j ⫽

再

G ⫺R i⫺1,j⫹1 )⫹e i⫹1,j⫺1 (R G i, j ⫺R i⫹1,j⫺1 )⫹e i⫹1,j⫹1 (R i, j ⫺R i⫹1,j⫹1 ) 兴 /(e i⫺1,j⫺1 ⫹e i⫺1,j⫹1 ⫹e i⫹1,j⫺1 ⫹e i⫹1,j⫹1 ) is the weighted sum of differences between R G i, j and the neighboring pixels of the red channel. It can be rearranged as RG i, j ⫺(e i⫺1,j⫺1 R i⫺1,j⫺1 ⫹e i⫺1,j⫹1 R i⫺1,j⫹1 ⫹e i⫹1,j⫺1 R i⫹1,j⫺1 ⫹e i⫹1,j⫹1 R i⫹1,j⫹1 )/(e i⫺1,j⫺1 ⫹e i⫺1,j⫹1 ⫹e i⫹1,j⫺1 ⫹e i⫹1,j⫹1 ), which is the value of the high frequency in the region of Fig. 2共a兲. Therefore Eq. 共16兲 can be the sum of the value of the low frequency and the value of the high frequency. The ␴ R/G i, j is the local standard deviation of log scale color ratios in Eq. 共14兲, and is evaluated as

关 log共 R i⫺1,j⫺1 /G i⫺1,j⫺1 兲兴 2 ⫹ 关 log共 R i⫺1,j⫹1 /G i⫺1,j⫹1 兲兴 2 ⫹ 关 log共 R i⫹1,j⫺1 /G i⫹1,j⫺1 兲兴 2 ⫹ 关 log共 R i⫹1,j⫹1 /G i⫹1,j⫹1 兲兴 2 4

2 ⫺ 共 m R/G i, j 兲

冎

1/2

共17兲

,

R/G where 0⬍ ␴ R/G i, j ⬍1, and m i, j is obtained from Eq. 共15兲.

3.4 Minus-Weighted Sum Based on the External Divide in the Green Channel The interpolation of the green channel is written as R R G i⫺1,j ⫹w i,R j⫺1 G i, j⫺1 ⫹w i⫹1,j G i⫹1,j G i, j ⫽ 共 w i⫺1,j R R ⫹w i,R j⫹1 G i, j⫹1 兲 / 共 w i⫺1,j ⫹w i,R j⫺1 ⫹w i⫹1,j ⫹w i,R j⫹1 兲

⫹ 共 f i⫺1,j G i⫺1,j ⫹ f i, j⫺1 G i, j⫺1 ⫹ f i⫹1,j G i⫹1,j ⫹ f i, j⫹1 G i, j⫹1 兲 ,

共18兲

locations G1 and G2, the process is the same as at location B except that two more original pixels are added. The interpolation of the blue channel can be performed in the same manner as the interpolation of the red channel. The updating process of the green channel at location R is presented in Eq. 共5兲, and the updating process at location B is similar. 4 Experimental Results The performance of the proposed algorithm was tested with simulated images 共‘‘Window,’’ ‘‘Sails,’’ and ‘‘Lighthouse’’ of 384⫻256 size; ‘‘Flowergarden’’ of 352⫻240 size; and ‘‘Boat’’ and ‘‘Houses’’ of 768⫻512 size兲 with an RGB

where f i, j is given by f i⫹n, j⫹m ⫽

2 m G␴ G

共 R i, j ⫺R i⫹2n, j⫹2m 兲 ,

共19兲

where (n,m)⫽(⫺1,0), 共0,⫺1兲, 共1,0兲, 共0,1兲. The m G is the mean of the green channel and ␴ G is the standard deviation of the green channel. After using the Eq. 共19兲, f i⫹n, j⫹m G i⫹n, j⫹n in Eq. 共18兲 is rearranged as the (2/m G ␴ G )G i⫹n, j⫹m (R i, j ⫺R i⫹2n, j⫹2m ), which is the weighted sum of the differences of the red channel, R i, j ⫺R i⫹2n, j⫹2m . Thus, Eq. 共18兲 is formed of the sum of the value of the low frequency and the value of the high frequency. 3.5 Algorithm The proposed color interpolation algorithm is illustrated in Fig. 4. Missing channel values are interpolated, then the updating process continues until the termination criterion is fulfilled. The interpolation of the green channel at location R is explained in Eq. 共18兲, and the interpolation at location B is similar to that at location R. The interpolation of the red channel at location B is shown in Eqs. 共9兲 and 共14兲. At 38

Optical Engineering, Vol. 43 No. 1, January 2004

Fig. 4 Block diagram of the proposed algorithm.

Park and Kang: Color interpolation with variable color ratio . . .

Fig. 5 Original images of (a) ‘‘Window,’’ (b) ‘‘Sails,’’ (c) ‘‘Lighthouse,’’ (d) ‘‘Flowergarden,’’ (e) ‘‘Boat,’’ and (f) ‘‘Houses.’’

Fig. 6 Partially magnified ‘‘Window’’ images (magnification ratio ⫽500%): (a) original image and images resulting from use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

Fig. 7 Partially magnified ‘‘Sails’’ images (magnification ratio ⫽500%): (a) original image and images resulting from the use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

Optical Engineering, Vol. 43 No. 1, January 2004

39

Park and Kang: Color interpolation with variable color ratio . . .

Fig. 8 Partially magnified ‘‘Lighthouse’’ images (magnification ratio ⫽500%): (a) original image and images resulting from use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

Fig. 10 Partially magnified ‘‘Boat’’ images (magnification ratio ⫽250%): (a) original image and images resulting from use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

Fig. 9 Partially magnified ‘‘Flowergarden’’ images (magnification ratio⫽500%): (a) original image and images resulting from use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

Fig. 11 Partially magnified ‘‘Houses’’ images (magnification ratio ⫽250%): (a) original image and images resulting from use of (b) the bilinear interpolation algorithm, (c) the edge-adaptive color interpolation algorithm, and (d) the proposed color interpolation algorithm.

40

Optical Engineering, Vol. 43 No. 1, January 2004

Park and Kang: Color interpolation with variable color ratio . . . Table 1 The PSNR comparison of conventional algorithms and the proposed algorithm.

Table 3 The MPSNR comparison of conventional algorithms and the proposed algorithm.

PSNR (dB)

Bilinear

Edge Adaptive

Proposed

MPSNR (dB)

Bilinear

Edge Adaptive

Proposed

‘‘Window’’ ‘‘Sails’’ ‘‘Lighthouse’’

23.7611 23.0814 20.6874

25.8186 26.4530 25.1089

27.4929 28.3378 27.6471

‘‘Window’’ ‘‘Sails’’ ‘‘Lighthouse’’

14.9927 14.1022 13.5584

17.4873 16.9740 16.3786

20.7617 20.4583 20.1278

‘‘Flowergarden’’ ‘‘Boat’’

16.3568 22.4635

19.1836 26.1536

22.0878 28.5277

‘‘Flowergarden’’ ‘‘Boat’’

13.1693 14.8526

16.1368 16.5807

19.2709 21.5702

‘‘Houses’’

18.7817

24.4967

28.3892

‘‘Houses’’

15.2092

17.9093

22.8847

Figures 6 –11 have four partially magnified images: the original image and the resulting images using the bilinear interpolation algorithm, the edge-adaptive color interpola-

tion algorithm16 without color correction and enhancement, and the proposed algorithm. The edge-adaptive color interpolation algorithm16 causes color artifacts along the edges during the color correction, thus initially interpolated the image without color correction and enhancement was used for the comparison in this paper. The bilinear interpolation algorithm oversmoothed the image resulting in color artifacts along the edges and in the detail of the image. Figures 6共b兲–11共b兲 show severe color artifacts and blurred edges. Cross-channel correlation was considered more in the edge-adaptive color interpolation algorithm. However, Figs. 6共c兲–11共c兲 still had color artifacts along the edges and in the detail of the images. The proposed color interpolation algorithm showed sharper edges, less color artifacts than conventional algorithms. For each algorithm of each image, the PSNR is tabulated in Table 1 and the NCD is tabulated in Table 2. The images were improved over 1 dB in the PSNR compared with conventional algorithms. Since the improvement in the proposed algorithm can be more effectively shown along the edges, the MPSNR and the MNCD were also used. The MPSNR and the MNCD are the PSNR and the NCD in the edge regions of the images. If a pixel is in the edge region, the PSNR and the NCD were calculated. If the pixel was not in the edge region, the PSNR and the NCD were not computed. The edges of the image were taken by thresholding the gradient magnitude of the image, by which the edge map was drawn. We used the same method to draw the edge map as Hur and Kang utilized to define the MISNR the integrated SNR at the edge and the MNCD in Ref. 17. The MPSNR is tabulated in Table 3 and the MNCD is tabulated in Table 4. Since the proposed algorithm showed large improvements along the edges and in the detail of the image, the increases in the MPSNR and the MNCD were larger. The images were improved over 3 dB in the MPSNR compared with

Table 2 The NCD comparison of conventional algorithms and the proposed algorithm.

Table 4 The MNCD comparison of conventional algorithms and the proposed algorithm.

NCD⫻10⫺2

MNCD⫻10⫺2

Bilinear

Edge Adaptive

Proposed

Bayer pattern. Original images are presented in Fig. 5. The peale SNR 共PSNR兲 and the normalized color difference27 共NCD兲, which is an objective measure of the perceptual error between two color images, were used to measure the performance of the proposed algorithm quantitatively. The PSNR was defined in decibels as PSNR⫽10 log10

2552 ⫻N 储 xˆ ⫺x 储 2

,

where N is the total number of the pixels in the image, x is the original image, and xˆ is the interpolated image. We computed the NCD in the L * a * b * color space, which is evaluated by using the following equation: NCD⫽

M 兺 i⫽1 兺 Nj⫽1 储 ⌬E Lab 储 M * 储 兺 i⫽1 兺 Nj⫽1 储 E Lab

,

where ⌬E Lab is the perceptual color error between two color vectors and is defined as the Euclidean distance between them, and is given by ⌬E Lab ⫽ 关共 ⌬L * 兲 2 ⫹ 共 ⌬a * 兲 2 ⫹ 共 ⌬b * 兲 2 兴 1/2, where ⌬L * , ⌬a * , and ⌬b * are the differences in the L * , a * , and b * components. The magnitude of the pixel vector * , and is given in the L * a * b * of the original image is E Lab by

* ⫽ 关共 L * 兲 2 ⫹ 共 a * 兲 2 ⫹ 共 b * 兲 2 兴 1/2. E Lab

‘‘Window’’ ‘‘Sails’’ ‘‘Lighthouse’’ ‘‘Flowergarden’’ ‘‘Boat’’ ‘‘Houses’’

Bilinear

Edge Adaptive

Proposed

7.4761 4.9973

5.4323 3.6594

4.6056 2.9893

‘‘Window’’ ‘‘Sails’’

10.9146 8.1147

8.3645 5.0945

6.4608 4.1894

7.3014 11.2620 5.2415

5.0736 8.4285 3.2117

3.9235 6.5497 2.4151

‘‘Lighthouse’’ ‘‘Flowergarden’’ ‘‘Boat’’

10.2178 21.2104 8.4892

7.3589 15.9753 4.6241

4.9119 12.4450 3.1884

8.3573

4.2357

2.8159

‘‘Houses’’

11.5220

5.4897

3.2716

Optical Engineering, Vol. 43 No. 1, January 2004

41

Park and Kang: Color interpolation with variable color ratio . . . Table 5 Run time comparison of conventional algorithms and the proposed algorithm.

Bilinear

Edge Adaptive

Proposed (Iteration Steps)

18 18

330 330

1880(2) 3120(3)

‘‘Lighthouse’’ ‘‘Flowergarden’’ ‘‘Boat’’

18 17 110

330 280 1490

4310(4) 1740(2) 6570(2)

‘‘Houses’’

110

1490

6570(2)

Run Time (ms) ‘‘Window’’ ‘‘Sails’’

conventional algorithms. Color artifacts along the edges and in the detail of the image that still existed when using the conventional algorithms were removed in the proposed algorithm. To show the computational burden of the presented methods, run times are tabulated in Table 5. The iteration steps at which the proposed algorithm converged are also noted in the Table 5. The experiment was performed on the PC equipted with the Pentium III CPU 共800 Mhz兲. The threshold T in Eq. 共6兲 was experimentally determined as 10 by comparing the color interpolated image’s mean square error 共MSE兲 to the original image with T for the sample images. The MSE had a minimum value when T was between 9 and 11, as shown in Fig. 12. Furthermore, the color-interpolated images did not look different in the visual performance if T had a value around 10. The constant k in Eq. 共7兲 was experimentally set to 1.6 for ‘‘Window,’’ 1.3 for ‘‘Sails,’’ 1.4 for ‘‘Lighthouse,’’ and 1.0 for the other images. The criterion 储 x n⫹1 ⫺x n 储 2 / 储 x n 储 2 ⬍10⫺5 was used to terminate the iteration, where x n was the image that was updated n times. 5

Conclusions

A color interpolation algorithm with a variable color ratio considering cross-channel correlation was proposed in this paper. The color ratio rule was applied to the edges and in the high-frequency detail of the image without causing color artifacts, for the color ratio was enabled to vary according to the color information at the pixel location. By adopting the color ratio of weighted channel sums instead of the weighted sum of color ratios, the averaged color ratio was changed less abruptly. During the updating process of the green channel, the color ratio that varied less sharply was chosen or the color ratios were incorporated according

Fig. 12 MSE versus T for the sample images. 42

Optical Engineering, Vol. 43 No. 1, January 2004

to the weight that reflected cross-channel correlation. The green value updated using the color ratio and the initially interpolated value of the green channel were iteratively combined, where the updating process was controlled according to the deviation of the color ratios. For the interpolation of the red and blue channels, the interpolated value using the color ratio rule was mixed with the interpolated value of a single channel. The larger the local mean of the log scale color ratios of the chrominance channel over the green channel became, the larger was the interpolated value of a single chrominance channel incorporated into the interpolated value using the color ratio rule. With the minus weighted sum based on the external divide, the interpolated value was not limited to the maximum or the minimum value in the neighboring pixels, and it could be composed of the sum of the low frequency and the high frequency. A 3⫻3 support region was adopted for the interpolation and the updating process of the green channel, the interpolation of the red channel at location B, and the interpolation of the blue channel at location R. In case of the interpolation of the red and blue channels at locations G1 and G2, a 5⫻5 support region was adopted. For the interpolation of the red and blue channels, a modified edge indicator function was utilized. The modified edge indicator function had three absolute differences for each directional derivative and was calculated by using only the pixels of the green channel. Experiments were carried out with various images. Using the proposed algorithm, a 1 dB was improved in the PSNR, and 3 dB was improved in the MPSNR compared with conventional algorithms, and color artifacts were removed, especially along the edges and in the detail of the image. Simulation results of the proposed algorithm indicated that color interpolation algorithm with variable color ratio considering cross-channel correlation outperformed conventional algorithms with respect to both objective and subjective criteria.

Acknowledgment This work was supported by the Korea Science and Engineering Foundation through the Biometrics Engineering Research Center at Yonsei University.

References 1. B. E. Bayer, ‘‘Color imaging array,’’ U.S. Patent No. 3 971 065 共Jul. 1976兲. 2. D. R. Cok, ‘‘Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal,’’ U.S. Patent No. 4 642 678, 共Feb. 1987兲. 3. J. E. Adams, Jr., ‘‘Interactions between color plane interpolation and other image processing functions in electronic photography,’’ Proc. SPIE 2416, 144 –151 共1995兲. 4. H. Sugiura, K. Asakawa, and J. Fujino, ‘‘False color signal reduction method for single-chip color video cameras,’’ IEEE Trans. Consum. Electron. 40共2兲, 100–106 共1994兲. 5. N. Ozawa, ‘‘Chrominance moire reduction using CCM signal interpolation single-chip color video camera,’’ ITE J. 46共2兲, 210–216 共1992兲. 6. M. Yoneyama et al., ‘‘Development to single chip digital camera with high picture quality,’’ Proc. ITE Annu. Conv. 8共5兲, 119–120 共1994兲. 7. H. Sugiura et al., ‘‘Luminance signal processing for single chip CCD color video cameras,’’ ITE J. 48共2兲, 210–216 共1994兲. 8. R. H. Hibbard, ‘‘Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,’’ U.S. Patent No. 5 382 976 共Jan. 1995兲. 9. C. A. Laroche and M. A. Prescott, ‘‘Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,’’ U.S. Patent No. 5 373 322 共Dec. 1994兲. 10. J. E. Adams and J. F. Hamilton, Jr., ‘‘Adaptive color plane interpolation in single color electronic camera,’’ U.S. Patent No. 5 506 619 共Apr. 1996兲.

Park and Kang: Color interpolation with variable color ratio . . . 11. J. E. Adams, Jr., ‘‘Design of practical color filter array interpolation algorithms for digital cameras,’’ Proc. SPIE 3028, 117–125 共1997兲. 12. J. F. Hamilton, Jr. and J. E. Adams, ‘‘Adaptive color plane interpolation in single sensor color electronic camera,’’ U.S. Patent No. 5 629 734 共May 1997兲. 13. H. Zen et al., ‘‘A new digital signal processor for progressive scan CCD,’’ IEEE Trans. Consum. Electron. 44共2兲, 289–296 共1998兲. 14. H. Kim, J.-Y. Kim, S. H. Hwang, I.-C. Park, and C.-M. Kyung, ‘‘Digital signal processor with efficient RGB interpolation and histogram accumulation,’’ IEEE Trans. Consum. Electron. 44共4兲, 1389–1395 共1998兲. 15. T. Kuno, H. Sugiura, and N. Matoba, ‘‘New interpolation method using discriminated color correction for digital still cameras,’’ IEEE Trans. Consum. Electron. 45共1兲, 259–267 共1999兲. 16. R. Kimmel, ‘‘Demosaicing: image reconstruction from color CCD samples,’’ IEEE Trans. Image Process. 8共9兲, 1221–1228 共1999兲. 17. B. S. Hur and M. G. Kang, ‘‘Edge-adaptive color interpolation algorithm for progressive scan charge-coupled device image sensors,’’ Opt. Eng. 40共12兲, 2698 –2708 共2001兲. 18. E. Chang, S. Cheung, and D. Y. Pan, ‘‘Color filter array recovery using a threshold-based variable number of gradients,’’ Proc. SPIE 3650, 36 – 43 共1999兲. 19. D. R. Cok, ‘‘Signal processing method and apparatus for sampled image signals,’’ U.S. Patent No. 4 630 307 共1984兲. 20. J. E. Adams, Jr., ‘‘Design of practical color filter array interpolation algorithms for digital cameras, part 2,’’ in Proc. IEEE Int. Conf. Image Processing, Vol. 1, pp. 488 – 492 共Oct. 1998兲. 21. J. W. Glotzbach, R. W. Schafer, and K. Illgner, ‘‘A method of color filter array interpolation with alias cancellation properties,’’ in Proc. IEEE Int. Conf. Image Processing, Vol. 1, pp. 141–144 共Oct. 2001兲. 22. D. Taubman, ‘‘Generalized wiener reconstruction of images form color sensor data using a scale invariant prior,’’ in Proc. IEEE Int. Conf. Image Processing, Vol. 3, pp. 801– 804 共Sep. 2000兲. 23. H. J. Trussell and R. E. Hartwig, ‘‘Methamatics for demosaicking,’’ IEEE Trans. Image Process. 11共4兲, 485– 492 共2002兲. 24. J. Mukherjee, R. Parthasarathi, and S. Goyal, ‘‘Markov random field processing for color demosaicing,’’ Pattern Recogn. Lett. 22, 339–351 共2001兲. 25. B. K. Guntruk, Y. Altunbasak, and R. M. Mersereau, ‘‘Color plane interpolation using alternating projections,’’ IEEE Trans. Image Process. 11共9兲, 997–1013 共2002兲. 26. A. M. Tekalp, Digital Video Processing, Prentice Hall PTR, Upper Saddle River, NJ 共1995兲. 27. L. Khriji, F. A. Cheikh, and M. Gabbouj, ‘‘High-resolution digital resampling using vector rational filters,’’ Opt. Eng. 38共5兲, 893–901 共1999兲.

Sang Wook Park received his BS and MS degrees in electrical and electronic engineering in 2001 and 2003, respectively, from Yonsei University, Seoul, Korea. He is currently a research staff member with LG Electronics Co. His interests are color interpolation of CCD images and enhancement.

Moon Gi Kang received his BS and MS degrees in electronics engineering from Seoul National University, Korea, in 1986 and 1988, respectively, and his PhD degree in electrical engineering from Northwestern University in 1994. He was an assistant professor at the University of Minnesota, Duluth, from 1994 to 1997, and since 1997 he has been with the Department of Electronic Engineering, Yonsei University, Seoul, Korea, where he is currently an associate professor. His current research interests include image and video filtering, restoration, enhancement, and reconstruction. He has published more than 100 technical papers in these areas. Dr. Kang is a member of IEEE and SPIE. He was on the editorial board of the IEEE Signal Processing Magazine, the editor of SPIE Milestone Series volume on CCD and CMOS imagers, the guest editor of the IEEE SPM Special Issue on Superresolution Image Reconstruction (May 2003), and a reviewer for the IEEE Transactions on Image Processing. He has served on the technical program and steering committees of several international conferences and as the associate editor for the Journal of Broadcast Engineering and the Journal of IEEK (the Institute of Electronics Engineers of Korea). He received 2002 HaeDong foundation Best Paper Award and the 2000 Award of Teaching Excellence from the School of Electrical and Electronic Engineering, Yonsei University.

Optical Engineering, Vol. 43 No. 1, January 2004

43