Optimal Recovery Demosaicing∗ D. Darian Muresan Cornell University Ithaca, NY. 14853, USA email: [email protected]

Thomas W. Parks Cornell University Ithaca, NY. 14853, USA email: [email protected]

ABSTRACT Digital color images from single chip digital cameras are obtained by interpolating a color filter array. Color information is encoded by means of a CFA, which contains different color filters (i.e. red green and blue), placed in some pattern. The resulting sparsely sampled images of the three-color planes are interpolated to obtain dense images of the three-color planes and thus the complete color image. Interpolation usually introduces color artifacts due to the phase shifted, aliased signals introduced by the sparse sampling of the CFAs. In this paper we discuss a non-linear interpolation scheme based on edge information, that produces better visual results than those obtained by linear and other published interpolation methods. KEY WORDS Demosaicing, Interpolation, Color Filter Arrays

1

Introduction

Due to hardware limitations, single CCD arrays in digital cameras do not capture the full red, green and blue color planes. Instead, they capture a sparsely sampled image of each of the color planes and interpolation is then used to reconstruct the original colors. In this paper we analyze the reconstruction of a single digital color image from the information provided by the Bayer CFA of Fig. 1 In previous work [1] we considered the problem of gray scale image interpolation from an adaptive optimal recovery point of view. We showed how a training set S determines a quadratic signal class and how to use this signal class to perform image interpolation. In this paper we extend our gray scale interpolation ideas to CFA interpolation and through examples show that the new interpolation scheme outperforms linear and other published interpolation methods. This paper is organized as follows. In section 2 we review the theory of optimal recovery as it applies to gray scale image interpolation [1]. In section 3 we present a unified view of some of the non-linear demosaicing methods in the current literature [3, 5, 4]. Finally, in section 4 we present our interpolation al∗ This

dak

work was supported by NSF MIP9705349, TI and Ko-

G

R

G

R

G

R

B

G

B

G

B

G

G

R

G

R

G

R

B

G

B

G

B

G

G

R

G

R

G

R

B

G

B

G

B

G

Figure 1. Bayer Array.

gorithm, which is a hybrid of different interpolation ideas. We conclude with section 5 where we present our results and provide some final thoughts on future work.

2

Optimal Recovery

We now review the interpolation method presented in [1], which is primarily an interpolation method for gray scale images. Locally, at pixel location y we model the image as belonging to a certain ellipsoidal signal class K (1) K = {x ∈ Rn : xT Qx ≤ ²} where Q is derived from a training set or may be assumed known. Vector x is any subset of the image containing the missing pixel y. Vector x is chosen such that any L linear functionals (Fi , i = 1, . . . , L) of x are assumed known. If we note the actual values of the functionals by fi we have Fi (x) = fi . In this paper we assume that fi are pixel values of the decimated image. The known functionals Fi , in the local image, determine a hyper-plane X . The intersection of the hyper-plane and ellipsoid is a hyper-circle in X . The intersection depends upon the known functionals of the local image and we call it Cx . The optimal recovery problem is to select the value in X which is a best approximation over all x ∈ Cx . We want to minimize y − y| δ = max |ˆ xT Qx

xTQ2x=c

xTQ1x=c

Figure 3. Image patch (left) and largest principal component of S (right).

Figure 2. Training set S and contours corresponding to xT Q1 x = c and xT Q2 x = c. The training set S is better represented by the contour corresponding to xT Q1 x = c than the one corresponding to xT Q2 x = c.

side of Fig. 3 we have a 20 × 20 image containing an edge. Matrix S is formed by taking all the 3×3 patches from the image. The largest principal component of S, which corresponds to the largest stretch in the ellipsoid, is shown on the right side of Fig. 3. Notice how this image vector retains the local edge direction.

3 The Chebyshev center achieves this minimization. The Chebyshev center has been shown to be the minimum Q-norm signal on the hyper-plane determined by the known samples. The solution to this problem is wellknown: see [6]. If the collection of known functionals is fi the minimum norm signal is u ¯. Signal u ¯ is the unique signal in X with the property, k¯ ukQ =

inf

Fi (x)=fi

kxkQ

(2)

and the optimal estimate, yˆ, is the pixel of the optimal image u ¯. From [6] the optimal vector u ¯ belonging to class K and satisfying the given functionals Fi is a linear combination of the representors φi of the given functionals Fi L X u ¯= αi φi (3) i=1

The challenge is to design a Q for which the class K is a proper representation of the local signal. Assume that a given training set S is to be used for learning our matrix Q. Form matrix S, of dimensions m × n, using the training vectors as columns (m is the dimension of the training vectors and n is the number of vectors in S). These training signals lie in a particular region of the space which we assume to be representative of the signal we are trying to interpolate. Our aim is to find a Q such that the ellipsoid

Demosaicing Review

The Bayer array (Fig. 1) contains more green (or luminance) pixels than red or blue in order to provide high spatial frequency in luminance at the expense of chrominance signals. Our task is to interpolate each of the R, G and B planes. The most basic idea is to independently interpolate the R, G, and B planes using linear interpolation. This type of interpolation introduces serious aliasing artifacts. The demosaicing approach of [2] improves the interpolation of the red and blue colors by adding high-frequency information from the green image to the red and blue images. It also uses the green highfrequency information to estimate the aliasing in the red and blue images, thus providing a means of reducing the amount of aliasing in the reconstructed image. An improvement over linear separable methods is to interpolate the ratios of red to green and blue to green, and to interpolate along edges not across them. Looking at Fig. 4, let Ei denote the likelihood that pixel P5 and pixel Pi belong to the same object. In other words, if P5 and Pi are part of the same object then Ei is close to one, otherwise it’s close to zero. With this Ei , the improved interpolation algorithm would then be: • First, interpolate the greens using only the green information: G5 =

x Qx = constant T

is representative of the training set S as shown in Fig. 2. In [1] we showed that Q = (SS T )−1 . To understand the directionality properties of this method we look at the following example. On the left

E2 G2 + E4 G4 + E6 G6 + E8 G8 E2 + E4 + E6 + E8

(4)

• For the blues, we proceed in two steps. First interpolate the missing blues at the red locations B5 = G5

B1 B3 B7 B9 E1 G + E3 G + E7 G + E9 G 1 3 7 9

E1 + E3 + E7 + E9

(5)

P1

P2

P4

P5

P7

P8

P3

With the above derivatives, [3] defines function Ei , for red, green and blue, as:

P6

1 , Ei = p 1 + D(P5 )2 + D(Pi )2

P9

Figure 4. Pixel P5 is the pixel that we are trying to interpolate.

and second, interpolate the missing blues at the green locations. B5 = G5

B2 B4 B6 B8 E2 G + E4 G + E6 G + E8 G 2 4 6 8

E2 + E4 + E6 + E8

(6)

where D is the derivative in the direction of Pi . As two examples, E6 = (1 + Dx (P5 )2 + Dx (P6 )2 )−1/2 and E3 = (1 + Dxd (P5 )2 + Dxd (P3 )2 )−1/2 . Next, if the ratio blue to green is constant within an object, so must be the ratio green to blue. If the ratio green to blue is constant within an object, then locally the ratio green to blue must be the average of the neighboring ratios. The green values are then adjusted appropriately. Of course, this offsets the original blue to green ratio. The authors of [3] go back and fourth three times correcting for the ratio rule of both blue and red. The correction step of [3] is approximately the following. • Repeat three times:

• Interpolate red the same way we did blue.

• Correct the Green values to fit the green over blue ratio test

All that is left to do now, is to choose a proper Ei function. In [5] the function Ei was defined as follows: 1). For green (i.e. we are trying to interpolate the green at pixel P5 ):    E2 = E8 = 1; E4 = E6 = 0 if |G4 − G6 | > T   |G2 − G8 | < T  |G4 − G6 | < T E2 = E8 = 0; E4 = E6 = 1 if   |G  2 − G8 | > T   E2 = E4 = E6 = E8 = 1 else

GB 5 = B5 GR 5 = R5

E2 = E8 = 1; E4 = E6 = 0 if P2 and P8 are blue E2 = E8 = 0; E4 = E6 = 1 if P4 and P6 are blue

In [3], function E is enhanced by using gradients in a more sophisticated way. From Fig. 4 we define our derivatives at pixel P5 , in the x, y, x − diagonal and y − diagonal directions as follows: P4 − P 6 2 P3 − P 7 √ Dxd (P5 ) = 2 2

P2 − P 8 2 P1 − P 9 √ Dyd (P5 ) = 2 2 Dy (P5 ) =

Notice that differences are always from the same color plane. If pixel P5 is a green pixel then so is P1 , P3 , P7 and P9 . To obtain the derivatives in the diagonal directions at a green pixel, we can do a little better by defining our diagonal derivatives as: ¯ ¯ ¯¾ ½¯ ¯ P3 − P 5 ¯ ¯ P 7 − P 5 ¯ ¯ ¯ ¯ ¯ √ , Dxd (P5 ) = max ¯ √ 2 ¯ ¯ 2 ¯ ¯ ¯ ¯¾ ½¯ ¯ P1 − P 5 ¯ ¯ P 9 − P 5 ¯ ¯ ¯ ¯ ¯ √ Dyd (P5 ) = max ¯ √ , 2 ¯ ¯ 2 ¯

E2 + E4 + E6 + E8 G4 G6 G8 2 E2 G R2 + E4 R4 + E6 R6 + E8 R8

E2 + E4 + E6 + E8

(8)

(9)

• Correct the Blue and Red values via the ratio rule P Bi Ei (10) B5 = G5 P Gi , with i 6= 5 Ei P Ri Ei (11) R5 = G5 P Gi , with i 6= 5 Ei

E1 = E3 = E7 = E9 = 1;

Dx (P5 ) =

G4 G6 G8 2 E2 G B 2 + E4 B 4 + E6 B 6 + E8 B 8

and average between the Blue and Red interpolation results GB + GR 5 G5 = 5 2

2). For red and blue (i.e. we are trying to interpolate the red or blue at location P5 ): ½

(7)

• End of loop. The enhancement steps of equations (8)-(11) tend to introduce some slight artifacts that can almost be classified as noise. In [3] this noise it taken out by a smoothing step.

4

Optimal Recovery Demosaicing

Our algorithm differs from [3] in two ways. First, we use a different method to interpolate the green plane. Second, instead of interpolating ratios we interpolate differences. The similarity between our algorithm and [3] is that we use the same weights Ei for the interpolation of the blue and red planes. More specifically, we interpolate the greens using optimal recovery and then

In interpolating the red and blue pixels we tried interpolating the ratios, but noticed that artifacts tend to be introduced in regions where the green values are very small. Experimentally, interpolating differences we obtained much better results. The complete interpolation algorithm is as follows: • First, interpolate the greens using optimal recovery, possibly taking a second pass using fine scale training vectors.

Figure 5. Coarse (left) and fine scale training vectors.

interpolate the differences between reds and greens and the blues and greens, using the same method as [3]. The key advantage to our approach is that the greens are interpolated better. The interpolation of green is done in two steps.

• For the blues, we proceed in two steps. First interpolate the missing blues at the red locations P i=1,3,7,9 Ei (Bi − Gi ) B5 = G5 + (12) E1 + E3 + E7 + E9

• Interpolate the greens using optimal recovery and coarse scale training vectors. The training set S contains coarse scale vectors, Fig. 5, taken from the neighborhood of the vector we are trying to estimate.

and second, interpolate the missing blues at the green locations. P i=2,4,6,8 Ei (Bi − Gi ) (13) B5 = G5 + E2 + E4 + E6 + E8

• Once we estimate the missing pixels, rerun the optimal recovery interpolation, but this time use a training set based on the fine scale vectors of Fig. 5. Experimentally, the second step is necessary only for high frequency patterns. It tends to improve the results, but it may not always be necessary.

• Interpolate red the same way we did blue. Next, add the correction step: • Repeat three times: • Correct the Green values to fit the green over blue ratio test P i=2,4,6,8 Ei (Gi − Bi ) B (14) G5 = B5 + E2 + E4 + E6 + E8 P i=2,4,6,8 Ei (Gi − Ri ) R (15) G5 = R 5 + E2 + E4 + E6 + E8

The selection of the training set S is key to the performance of optimal recovery interpolation. For the demosaicing case, training set S was obtained by selecting the nearest vectors (both in space and value) to the vector we were estimating. More specifically, the training set S was formed as follows.

and average between the Blue and Red interpolation results GB + GR 5 G5 = 5 2

• Decide on a window size for selecting the training set S. If the size is too large, local feature of the image are missed. Smaller window sizes seem to perform better than larger ones. Experimentally, a window of 15 × 15 pixels around the pixel we are estimating seems to perform well for a 256 × 256 image.

• Correct the Blue and Red values via the ratio rule P Ei (Bi − Gi ) P , with i 6= 5 (16) B5 = G5 + Ei P Ei (Ri − Gi ) P , with i 6= 5 (17) R5 = G5 + Ei

• In this window, which defines closeness in space, select all the possible training vectors. Call this set S. • Prune S by selecting only half the vectors. Take the half of the vectors that are closest to the vector we are trying to estimate, based on the known samples. This defines a closeness in value. The last step of the formation of the training set S is critical, especially at boundary regions of two different textures. For example, in the light house picture, at the boundary between the fence and the grass, if the last step of the selection of S is not performed, the fence pattern seems to be repeated into the grass.

• End of loop.

5

Results

We applied our demosaicing algorithm to the lighthouse, sails, window and statue images. Starting with the full color images we sampled them using the Bayer array pattern of Fig. 1 and interpolated back to full color images. The color images of [3] were obtained directly from the author’s web page in a lossless PPM

Table 1. Lighthouse (top) and Sails (bottom) PSNR values (db) Method Linear Glotzbach Kimmel OR Method Linear Glotzbach Kimmel OR

Red 23.73 28.02 32.79 34.62 Red 26.48 30.01 35.06 37.58

Green 27.76 31.09 36.62 37.59 Green 29.85 33.38 39.29 40.09

Blue 23.95 27.82 31.38 35.29 Blue 26.72 29.98 35.85 38.23

Table 2. Window (top) and Statue (bottom) PSNR values (db) Method Linear Glotzbach Kimmel OR Method Linear Glotzbach Kimmel OR

Red 26.62 29.28 32.16 35.65 Red 28.04 29.05 35.30 37.94

Green 29.98 32.43 34.69 38.11 Green 30.15 31.46 38.66 39.40

Blue 26.52 29.52 33.33 34.80 Blue 27.74 28.93 34.85 37.59

format. They reflect the author’s implementation and not our version of the author’s algorithm. The PSNR values for the reconstructed lighthouse and sails image are given in Table 1 and those of the window and statue images are given in Table 2. Although the PSNR is not necessarily a good measure of image quality, both tables reflect an average improvement of at least 2db over [3], at least 7db over [2] and an average improvement of at least 10db over linear interpolation. For our results (Fig. 6 and 7) we have the lighthouse image reconstructed using linear and our optimal recovery demosaicing. Notice how optimal recovery demosaicing almost entirely removes the color moir´e patterns. On the side of the house, Fig. 6, linear interpolation performs miserably, while optimal recovery demosaicing performs best. In this paper we included comparisons only with linear interpolation because the differences in the color images are hard to notice on gray scale prints. The interested reader may see the comparisons with Kimmel and Glotzbach at http://dsplab.ece.cornell.edu. There is an interesting observation to be made about the optimal recovery demosaicing result. In particular, notice that the optimal recovery demosaicing result has some color artifacts on the frame of the win-

dow: Fig. 6. The artifact patterns are coming from the house siding. In other words, the quadratic signal class generated around the window was heavily influenced by the house siding pattern. This is a place where choosing the correct training set for learning the quadratic class can be improved in future work.

6

Conclusion and Acknowledgments

In this paper we applied the optimal recovery interpolation model presented in [1] to the problem of CFA interpolation. Through examples we have shown an improvement over the linear interpolation models and other recently published directional interpolation algorithms [3] and [2]. Finally, we would like to thank Ron Kimmel for responding to all our email queries about his paper and to John Glotzbach for providing us with his Matlab code.

References [1] D. Darian Muresan and Thomas W. Parks, “Optimal Recovery Approach to Image Interpolation,” ICIP 2001, Greece, 2001. [2] Glotzbach, J.W.; Schafer, R.W.; IlIgner, K. , “A method of color filter array interpolation with alias cancellation properties,” Image Processing, 2001. Proceedings. 2001 International Conference on, Volume: 1 , pages: 141 -144, 2001. [3] Ron Kimmel, “Demosaicing: Image Reconstruction from Color CCD Samples,” IEEE Transactions on Image Processing, Vol. 8, No. 9, pp. 1221-1228, (September 1999). [4] J.E. Adams, Jr., “Design of Practical Color Filter Array Interpolation Algorithms for Digital Cameras,” Proceedings of SPIE, D. Sinha, ed. Vol. 3028, pp. 117-125, SPIE, Bellingham, WA. 1997. [5] D. R. Cok, “Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal,” U.S. Patent 4,642,678, 1987. [6] M. Golomb and H. F. Weinberger, “Optimal approximation and Error Bounds,” On Numerical Approximation, R. E. Langer ed., The University of Wisconsin Press, Maddison, pp. 117-190, 1959.

160

160

170

170

180

180

190

190

200

200

210

210

220

220

230

230

240

240 20

30

40

50

60

20

30

40

50

60

Figure 6. Lighthouse Zoom One: linear (left) and optimal recovery demosaicing (right).

170

170

180

180

190

190

200

200

210

210

220

220

230

230

240

240

250

250 110

120

130

140

150

160

110

120

130

140

150

160

Figure 7. Lighthouse Zoom Two: linear (left) and optimal recovery demosaicing (right).