Project: Hardware Implementation of Color Image Demosaicing for Camera Cellphones

Under Guidance of: Prof M Balakrishnan Ass. Prof. Prem Kalra

By: S Vijaya Raghavan 2004JVL2439

Associated with: Cypress Semiconductors Pvt. Ltd., Bangalore

What is Demosaicing

Nearest Neighbor Replication •Each interpolated output pixel is assigned the value of the nearest pixel in the input image. •The nearest neighbor can be any one of the upper, lower, left and right pixels. •An example is illustrated below in Figure 2 for a 3x3 block in green plane. Here we assume the left neighboring pixel value is used to fill the missing ones.

Bilinear Interpolation •For convenience, Bayer pattern is duplicated here

•Interpolation of green pixels : • Average of the upper, lower, left and right pixel values is assigned as the G value of the interpolated pixel G8 = (G3+G7+G9+G13) / 4 •Interpolation of red/blue pixels : • Interpolation of a red/blue pixel at a green position : • the average of two adjacent pixel values in corresponding color is assigned to the interpolated pixel. •Example : B7 = (B6+B8) / 2 ; R7 = (R2+R12) / 2 • Interpolation of a red/blue pixel at a blue/red position : • the average of four adjacent diagonal pixel values is assigned to the interpolated pixel. •Example : R8 = (R2+R4+R12+R14) / 4 ; B12 = (B6+B8+B16+B18) / 4

Smooth Hue Transition •Interpolation of green pixels : •same as in bilinear interpolation •interpolation of green pixels-- done before red/blue pixels • Interpolation of red/blue pixels : •try to impose a smooth transition in hue from pixel to pixel. •define blue "hue value" as : B/G. And red "hue value" as R/G •three different cases of blue pixel value interpolations • blue pixel value at green position + adjacent blue pixels are on left and right : e.g. B7 = G7 / 2 * (B6 / G6 + B8 / G8) • blue pixel value at green position + adjacent blue pixels are on top and bottom : e.g. B13 = G13 / 2 * (B8 / G8 + B18 / G18) • blue pixel value at the red postion : •e.g. B12 = G12 / 4 * (B6 / G6 + B8 / G8 + B16 / G16 + B18 / G18)

Smooth Hue Transition (log space) •Interpolation of green pixels : •same as in bilinear interpolation •interpolation of green pixels-- done before red/blue pixels • Interpolation of red/blue pixels in log space : •define blue "hue value": logB- logG , red "hue value" : logB- logG •three different cases of blue pixel value interpolations • blue pixel value at green position + adjacent blue pixels are on left and right : e.g. logB7 = logG7 + (logB6 - logG6 + logB8 - logG8) / 2 • blue pixel value at green position + adjacent blue pixels are on top and bottom : e.g. logB13 = logG13 + (logB8 - logG8 + logB18 - logG18) / 2 • blue pixel value at the red position : •e.g. logB12 = logG12 + (logB6 - logG6 + logB8 - logG8 + logB16 - logG16 + logB18 - logG18) / 4 •Finally, transfer all pixel values back into linear exposure space

Edge Sensing Algorithm I •Human visual systems are sensitive to edges in images •Non-adaptive interpolation algorithms often fail around edges •since they are not able to detect "edges“

•Interpolation of green pixels : •First, define two gradients, one in horizontal direction, the other in vertical direction, for each blue/red position. For instance, consider B8 : define two gradients as , where | . | denotes absolute value •Define some threshold value T •The algorithm then can be described as follows :

Edge Sensing Algorithm I • Choice of T depends on the images •T-- different optimum values from different neighborhoods •Common choice of T is •Algorithm now becomes :

• Interpolation of red/blue pixels : •Same as in Smooth hue transition interpolation in logarithmic space • Other methods may be used as well

Edge Sensing Algorithm II •A slightly different edge sensing interpolation algorithm •Interpolation of green pixels : •Same as in edge sensing algorithm I except that horzitonal and vertical gradients are defined differently. •Refer to figure below, to estimate G5 at B5, we define gradients as :

•Interpolation of red/blue pixels : •similar to bilinear interpolation algorithm except that color difference is interpolated instead of color itself.

Linear Interpolation with Laplacian second-order correction terms •Interpolation of green pixels : •Consider estimating the green value at a blue pixel •Find out G5 at B5 •Define horizontal and vertical gradients as :

•The algorithm is :

Linear Interpolation with Laplacian second-order correction terms •Interpolation of red/blue pixels : • Depending on the positions, we have three cases 1. find red/blue values at a green pixel where nearest neighbors are in the same column. •estimate R4 at G4, •R4 = (R1 + R7) / 2 + (G4 - G1 + G4 - G7) / 4 2. find red/blue values at a green pixel where nearest neightbors are in the same row •estimate R2 at G2, •R2 = (R1 + R3) / 2 + (G2 - G1 + G2 - G3) / 4

Linear Interpolation with Laplacian second-order correction terms •Interpolation of red/blue pixels : 3. find red/blue values at a blue/red pixel • estimate R5 at B5 •Define two diagonal (negative and positive) directions

•Now the interpolation is described as :

High Quality Linear Interpolation •Edges have much stronger luminance than chrominance components •Example: Compare red value to its estimate for a bilinear interpolation •If different there’s a sharp luminance change at that pixel •Correct a bilinearly interpolated green value by adding a portion of this estimated change ¿ ¿¿ •To interpolate G value at R location: where gradient of R at that location is αΔR i,j subscript B means bilinear interpolation

•To interpolate R value at G location:

¿ ¿¿ ¿ ¿¿

•To interpolate R value at B location: •Gradient correction Gains •Determine gain parameters using Weiner approach •Compute values that lead to minimum mean-square error interpolation

α= 1/2, β= 5/8, γ= 3/4

High Quality Linear Interpolation

Filter coefficients •Simple •Linear •Better than some nonlinear algorithms •Computational complexity is more

Discriminated Color Correlation •Use color signal correlation •Select image signal with higher correlation (horizontal/ vertical) •HIGH HOR. FREQ. => HIGH COR. VER. DIR. •HIGH VER. FREQ. => HIGH COR. HOR. DIR. •Improve resolution and reduce aliasing •In local region, variations in three signals are similar to one another

G3:R3=GLPF:RLPF G3=R3∗GLPF/RLPF

Discriminated Color Correlation •Interpolate G at R or B pixel •Calculate gradients

•Obtain G values using below formula

•Interpolate R and B signals at G position

Discriminated Color Correlation •Interpolate B at R pixel

•Similarly for R at B pixel

Freeman Demosaicing •

G sensor class is sampled at a higher rate than the R and B sensor classes

• 4. 5. 6.

ALGORITHM Apply bilinear interpolation separately to each sensor class The G image plane is now used directly as the output G image plane For R and B planes, result of bilinear interpolation is modified before output 1. Two difference planes R-G and B-G are created 2. Median filtering of R-G and B-G planes done to eliminate artifacts 3. R and B planes are created by adding the G plane to the median filtered R-G and B-G planes 4. In final output image, R and B values estimated from the median filtered R-G and B-G planes are used only at pixels where there is no R or B sensor value directly available

Freeman Demosaicing • • • •

• •

PROBLEM Straightaway implementation of this leads to bad performance in general Generally, R, G and B sensor classes have very different mean responses in images This algorithm is sensitive to differential scaling

SOLUTION Scaling R, G and B sensor planes to have the same mean before applying this algorithm and undoing this scaling afterwards produces much better results

Fourier Domain Demosaicing •Color demosaicing by estimating luminance and opponent chromatic signals in Fourier Domain

{Ci  x,y  } ,i∈ [ R,G,B ] ,  x,y  ∈N2 •One-color per pixel image can be written as sum of luminance and chrominance Chrominance= 3-D vector •Luminance =scalar •Luminance and chrominance are well-localized in spatial frequency domain

•Spatial multiplexed image

R  x,y =∑ Ci x,y  m i x,y 

•where

i

mR x,y =1cosπx1cosπy/4 mGx,y =1−cosπxcosπy/2 mBx,y =1−cosπx1−cosπy/4

Fourier Domain Demosaicing mi(x,y)= pi + mi~(x,y) •Thus •Hence we have: R  x,y =∑ piCi x,y ∑  x,y  ¿ i

i

•First part is Luminance => doesn’t depend on modulation function at each pixel •Second is Chrominance => has color information that is spatially sub-sampled

ALGORITHM

Fourier Domain Demosaicing Luminance Filter

•Chrominance Filter •Orthogonal filter to Luminance

Vector Color Filter Array Demosaicing •1. Solving for a color vector instead of plane-by-plane pixel interpolation •2. Replacing original color value at a pixel with value that matches better with other color dimensions •Pseudo-pixels: group neighboring Red, Green and Blue values together •Use column vector to represent Pseudo-pixels •Replace unknown color value with average of surrounding pseudo-pixel values •NOT arithmetic average – smearing •To Preserve edges, take vector median of pseudo-pixels to be missing color vector R 21 G 21 B 21 r ig h

R 15 : R 15 : R 15 : R 15 : R 2 7 : R 2 7 : R 2 7 : R 2 7 : G 2 1 : G 2 1 : G 16 : G 14 : G 2 1 : G 2 1 : G 2 8 : G 2 6 B 2 2 :B 20 :B 2 2 :B 20 :B 2 2 :B 20 :B 2 2 :B 20 r ig h [ ][ ]

Vector Color Filter Array Demosaicing •Vector Median is a vector A that minimizes the sum of L2 distances (Euclidean) to each point in the set. •Let {x1,x2, … ,xn} be set of K-dimensional vectors, then Vector Median is vector A that solves: N

argmin A ∑

∑ K

i= 1 k= 1



1/ 2

 Ak−xik 

2

here Ak and xik are k-th element of vector A and xi, respectively •Vector median has to be found by iterative means and can be approximated in a number of ways •“Exact” vector median is one that is accurate to some ε, where ε is smaller than the quantization error of imaging system

Variance Interpolation •Maintains sharpness of edges •Compute amount of variance ( difference between this pixel and its neighboring pixels) •Assume variance is similar in all three color bands •Requires temporary storage to hold 5 rows

Algorithm overview •For each pixel, determine the actual color value it stores (Refer Bayer pattern) •Now, compute the average value of the same color for this pixel from its corresponding neighboring pixels •Compute the “difference” between the actual and average color value •Compute the average value of the missing color values for this pixel from the neighboring corresponding pixel values -can use bilinear interpolation here •To each estimated missing color value add the “difference” value computed

Image Enhancement by Efficient Demosaicing •Two main stages •Adaptive initial interpolation •Correction of visual artifacts that remain in image •In whole algorithm has 4 steps as shown in figure below:

Image Enhancement by Efficient Demosaicing •Initial estimation (Initial interpolation) •Green Channel •Classify missing pixels of green channel into regions •Apply different interpolation schemes according to the classification •Have an estimate now of the location of highly problematic regions •Classify these regions further into: •Horizontal region: interpolate in horizontal direction •Vertical region: interpolate in vertical direction •Right direction is determined in the correction stage •Red/ Blue Channel •Interpolate these channels twice ( for both estimates of green channel) •Resulting two color images are passed to correction stage, color artifacts are reduced •Main key detecting artifacts: •Relative smoothness of chrominance component of each region (YIQ space)

Image Enhancement by Efficient Demosaicing •Green Estimation •Class A pixels (very smooth) •Average value of neighbors •Class B pixels •Interpolate using larger neighborhood •Neighbors of missing pixel+pixels from other two channels •Class C pixels (problematic pixels) •Neither neighbor pixels nor inter-channel pixels help in estimating unknown pixel •Caused due to aliasing and sub-sampling •Unknown value = average of left & right or average of upper & lower neighbors •Red/ Blue Estimation •Red-Green and Blue-Green values interpolated •Performed twice for each channel (Horizontal and Vertical regions) •YIQ Transformation •Considering I component, for each aliased region •Select smoother region based on relative smoothness of horizontal and vertical versions

Image Enhancement by Efficient Demosaicing •GREEN PIXEL •Class A pixels

¿ ¿ ¿¿

•Class B pixels

¿

¿

¿

•Class C pixels •Horizontal or Vertical average

•RED/ BLUE PIXEL G 2  R1−G1  ifR 1