OPTIMIZATION OF THE JOINT CODING/DECODING STRUCTURE C. Parisot, M. Antonini, M. Barlaud
S. Tramini, C. Latry, C. Lambert-Nebout
I3S laboratory - UPRES-A 6070,
Centre National d’Etudes Spatiales - CNES - Bpi 1421
2000 route des Lucioles , Bât. Algorithmes/Euclide F-06410 Biot (FRANCE)
18, Avenue Edouard Belin, F-31401 - Toulouse - Cedex 4 (FRANCE)
(parisot,am,barlaud)@i3s.unice.fr
[email protected], (Christophe.Latry, Catherine.Lambert)@cnes.fr
ABSTRACT This paper considers the estimation scenario where an original image has undergone blurring, noise corruption and compression. The authors consider these unwanted effects jointly in the estimation formalism they propose. The contribution of the paper aims to adress the joint coding/decoding formulation, involving a priori assumptions on the solution and knowledge of the imaging systems to account for effects due to acquisition noise and compression noise. Most of the authors’ construction is based on well-known techniques drawn from a variety of areas in modern signal processing, including optimization theory, wavelet decomposition, optimal bit allocation, inverse problem and bounded noise assumption.
1. INTRODUCTION Despite advances in the domain of source coding, the joint coding/decoding structure has been, to our knowledge, little investigated. Usually, the optimization of the compression is done separately: we optimize coder, decoder and then post-processing. In this paper, we focus on still-image coding/decoding structure for remote sensing applications (see Fig. 1). Here, the original image has undergone blurring, noise corruption and compression. The contribution of the article concerns the joint treatment of coding, decoding, denoising and deblurring in order to increase the performance of the complete acquisition/compression/restoration chain. The outline of this paper is as follows: section 2 describes the global acquisition/compression system; section 3 is devoted to the encoder optimization based on acquisition system a priori; in section 4 we propose a restoration process which accounts for acquisition and compression processes. This joint coding/decoding structure is examined in section 5, and shown to yield reduced artifacts and to preserve information (high frequency details). Finally, section 6 concludes the paper.
Fig. 1. Imaging system 2. PROBLEM STATEMENT: ACQUISITION/COMPRESSION SYSTEM The image acquisition process from a signal processing point of view is reduced to: Modulation Transfer Function (MTF) 1 , signal to noise ratio 2 and sampling grid [2, 3]. The radiance entering the instrument may be considered as a bidimensional continuous function that is convolved by the instrument point spread function , corrupted by noise and finally sampled according to a sampling grid. The limited size of spatial telescope optical apertures leads to a MTF weak value at Nyquist frequency and a small signal to noise ratio. The blur operator ! "
#$%& ! ' is assumed to be anti aliasing (the sampling grid is assumed to meet shannon sampling requirements), linear, symmetric and compact. Thus, the linear degradation relation connecting to )( , taking account of both blur and noise 3 , can be written as:
)(*+ ,$-.
(1)
The logical step would be to restore the blurry-noised /( image before compressing it. But the on board instruments’ cost constrains to defer complex and expensive processing 1 inverse
Fourier transform of the impulse response and acquisition noise are described in [1]. 3 This signal includes a noise with different origins and its standard deviation can be approximated by . 2 MTF
023/1 465879;:=@?@ABDCE4F5879;:
to ground. Thus, denoising/deblurring is not possible before compression. It is then important to take noise and blur into account in the encoding step (section 3).
G/H
is then compressed using the classical three step processus: signal decorrelation using a wavelet transform opG/L erator I J 4 K , I transformed data quantization and coding. Here, we focus on the particular case of uniform scalar quantization, then, the quantization function M is defined c f Y#`hiec ] where by: MONQPSRUTNDVXWRZY if V\[^]_Y#`badeg ` is the quantization step and Yj[kT . The decoder generally appplies the inverse operations for decoding the quantized image. The inverse quantization process is defined by: Mmlon#NTiRpPqNrYjWRtVs where Vu s [ ] Y,`Davec f Y,`whxec ] is known at the decoder. The quantization process entails a loss of information between the quantized data and the input signal. The quantization error can be assimilated to an additive noise and thus,
y { GL G L z| I h } ~ I
(2)
3. ENCODING METHOD
G
e
{d . Let Thus, for this section, we consider that eL and be respectively the variances of the 2 subband G GL of the wavelet decomposition of and . being a lowpass filter, it weakens the coefficients magnitude in the high frequency subbands.
Our basic idea is to use some weightings in the total distortion criterion in order to compensate this decrease. This way, the encoded image has almost the same frequency enG ergy distribution than the target image defined as in [3], section 4. The bit allocation criterion is
e ~` { w ` h ~ /v ` a
(3)
e
where and are respectively the distortion and the bit rate for each subband when quantized by ` and is the output target rate. The distortion in the subband is e proportional to the variance [4]. Thus, one easy way to e GL simulate the energy in the 2 subband of is to take e e { ; L . Unfortunately, those ratios can’t be computed from
since they are also related with the signal. To compute optimum theoretical weightings, we should only consider subbands in 4
¡=¢£¥¤¦Q§¡'¢£6¤o¨S¦ª©&«E£F¬8 ®;¦=§ ¯ °«E£F±;²¦³
which the frequency distribution of can be approximated by a constant. Such approximations are only possible with an accurate frequency decomposition. Consequently, as a sub-optimal solution, we will compute the ratios under the assumption that the image spectrum is uniform (same contribution of all frequencies). This is performed by generating a white noise ¹ with uniform spectrum and by computing ¹º result e of the econvolution of ¹ by the deconvolution filter. Let » and » º be respectively the variances of the
2
In this section, we consider the optimization of the encoding process according to the knowledge of the acquisition Modulation Transfer Function. The wavelet image coder chosen is $
E [4, 5] since it is a candidate for future CNES Earth observation missions [3]. Here, we do not take account of the acquisition noise.
GL
Fig. 2. Extract of Nimes restored by ´¶µ|·b¸ - CR = 20:1
subband of ¹ and ¹º . Then , the weightings are
e e { » º »
(4)
These weights quantify the impact of the blurring operator on the energy contained in each subband of the wavelet decomposition. They increase the importance of the highest frequencies in the compression process and consequently improve the details preservation. 4. DECODING METHOD We considerG in this section the problem of recovering uny known data from quantized observed data z , knowing f I and the quantizer M . Retrieving the acquisition system input image is impossible. The best we can do is retrieving frequency information not filtered by MTF . Thus, the principle is to use a target reference image that would be generated by a perfect instrument. A square sampling grid with sampling interval J 5 K ¼ n ½ leads to retrieve the frequencies belonging G G G G to the square defined by ¾Fa ½ )¿ À ½ /¿Áà¾Äa ½ /¿!À ½ )¿Á . Therefore, a perfect MTF would be the characteristic function of this square but the corresponding Point Spread Function would be a slowly decaying oscillating function. Thus, we define a target MTF as a slightly regularized characterG 6K istic G=Æ function G noted Å J . The desired solution is not but
{ Å
The restoration problems are usually tackled using classical image enhancement and image restoration techniques which 5 6
È
«Ç
is the sampling frequency of signal. satisfy Shannon conditions.
do not take compression and acquisition noise into account. The method proposed herein, is based on a previous frameG work [10], in which the estimated image s is given by:
G s {ÊÉEË;Ì
Ý Í Î/Ï @ Í ØFÚ !à á ~ G h à â ~ G f ã Ù Ð;ÑÒÓÔ_Õ/Öo× ÑÒ ÓÛÔÜÕ/× Þß n
(5)
Data driven term modeling - The first term, namely data driven term, takes the output coder data into account. e We introduce two weightings for each subband , en äå l ¥ and æ in order to respectively take account of the quantization noise characteristics and to minimize the mean quadratic error [6] in the case of non-orthogonal filters.
e G y e å l ¥ æ ~I a z=ì ä)í î/í8ï ¨ ¿ ë y y where z=ìð{ I ñdI$ò z . àQç G { éè8ê ~
(6)
Regularizing term - Adaptive smoothing - The second term contains a priori assumptions on the solution. It allows noise removal while preserving edges [7, 8].
e è ê ô¶õ L G
~ ö÷ó ö í V í2ø (7) e ~ÿ hÿ is assumed to be an even e function J 7 K of class ~P . Since the
G { Ó ~ ó Here ô õ Lù{ûú ~ üDýÃþ à
convex potential drawback of the ü function is non-differentiability, a solution consists in regularizing ü , using a{ Gaussian function þ [ ~P such that ~þ ýXü þ ýXü The following conditions on the potential function are pro posed: isotropic smoothing in homogeneous areas: Ø6Í ôÆ õ L ~ÿ /¿ ÿ { ; preservation of edges: Ø6Í ô õÛL ~%ÿ )¿ ÿ { ; ô õÛL ~%ÿ )¿ ÿ is strictly decreasG H ing in order to avoid instabilities. If ó , we suppose that data stands for noise, and then, we apply a strong smoothing by using a total variation ô¶õ L function ~ ü|R . G H If ó Ê , we suppose data corresponds to information, and we apply a light smoothing in order to preserve high frequency details.
Spatio-frequency a priori - Soft thresholding - The third term acts as G a soft thresholding [8], i.e as a penalty on coefficients I .
à G {u e è ~
õ ¨ L õ ~ I G 2í î/íEï å
(8)
is a convex potential function. As for the ô¶õ L funcG tion of Ó ~ regularizing term, we L adapt this function to the noise level by introducing a ¨ and å dependence. The noise is removed by removing coefficients that are insignifi cant relative to a threshold defined by õ ¨ L õ å for each sub band
õ ¨ L õ åà
Õ ¨
7
£ 2¦"!$#&%( )' *
«, .-0/ 1 Ç3+ 2 4 Ç576/Ç 8ÛÇ ³
Bounded noise a priori - Quantization error is bounded. According to $
E , for each subband , the coder y output z belongs to 96Y,` a c eÃf Y#` 9 and the quantiza-
tion range is ;: { 9 Y#` a c ebf Y,` h c e 9 J 8 K . Therec fore, global acquisition noise < can be assimilated to a nonstationary uniform noise. Thus, we can determine a range for the degradation noise due to acquitision y and compression, = { }bh+I>< . Knowing the coefficient z , the quantization step ` and some information on acquisition noise < (such as distribution, variance L...), we can estimate for each point a bound ? { ` /¿ hA@ for each reconstructed soà â G f ã G ~ I to term constraints all value lution site J 9 K . belong the range { ]_Y,` a$? f Y#` hA? ]
à â G f ã { ~
Þ6ß
è
ã á Ì á ~ G 2í î/íEï
Õ ¨ l ª n n á G { G where Ì ~ ~CB~ I aY,` $ a ? :
.
G
(9)
Decoding criterion - Suppose that a solution ~ s f ã s to this problem exists, it verifies the Euler equation (10) with Neumann boundaries condition. I ò and ò stand G for the adjoint integral operator. To solve this problem in ~ f ã , the proposed algorithm uses a fast search method described in [10]. 5. EXPERIMENTS Our experiments were performed on the 10 bits/pixel image "Nîmes" of size 512x512 pixels. This blurry-noised image was provided by the French Space Agency (CNES). MTF and on board signal to noise ratio are representative of the future remote sensing systems. Weightings defined in section 3 allow high frequency details preservation (see Fig. 2), and involve a significant PSNR gain at high compression ratios (about 0.4 dB for CR =20:1). We present on Fig. 3 the image restored by D · (a deblurring/denoising method developped and used by CNES [11]) and the image restored by ´¶µ ·Ã¸ (Optimal Decoding and Restoration Technique). In this example, we use a three level wavelet decomposition. Those first results show an appreciable improvement in visual quality, with high frequency details preservation and acquisition/compression noise removal, as well as PSNR gains ranging from 0.6 to 1 dB (see Table 1). 6. CONCLUSION This paper presents a global vision of a very important problem in image processing (coding, deblurring, noise removal and post-processing after compression). Our cod8 where 9R
¨ L
E
is G F
dependent according to the formula E
, bound of noise S in
¤o¨
Ó
K ML,N !&HJI O QP .
domain is defined as in [9]
Fig. 3. Extract of Nîmes (Left to Right: reference, TUWV(XZY[]\_^a`cb3da\fe + gY.hji , TUkV(X]Y[Z\l^ `cb3dm\le + n.opirq - CR = 6:1) ~ sjtvu txwy z|{~} u.s
p
CR
TUkV(X]Y [Z\l^ ±C`c²cb3³ dm\le
2.5 3 3.5 4 4.5 5 5.5 6
32.49 dB 32.47 dB 32.44 dB 32.40 dB 32.36 dB 32.33 dB 32.3 dB 32.2 dB
~
A ~3
7 ]v
(1) + gY.h´i
(1) + n.opirq
39.9 dB 39.7 dB 39.4 dB 39.05 dB 38.67 dB 38.3 dB 38.01 dB 37.7 dB
40.91 dB 40.7 dB 40.3 dB 39.92 dB 39.51 dB 39.2 dB 38.8 dB 38.3 dB
w ¡¢
C. Latry and B. Rougé, "SPOT 5 THR Mode", in SPIE conference on Earth Observing Systems III, vol. 3439, San Diego, USA, 1998.
Thanks to B. Rougé and G. Moury from CNES for their fruitful collaboration and their sound advice
(10)
C. Latry, B. Rougé, "Optimized Sampling for CCD Instruments: the Supermode Scheme", in IGARSS conference, Hawai, USA, 2000.
[3]
C. Lambert-Nebout, C. Latry, G. Moury, C. Parisot, M. Antonini and M. Barlaud, "On Board Optical Image Compression for Future High Resolution Remote Sensing Systems”, in SPIE conference, San Diego, USA, 2000.
[4]
C. Parisot, M. Antonini and M. Barlaud, "EBWIC: A Low Complexity and Efficient Rate Constrained Wavelet Image Coder”, in ICIP, Vancouver, Canada, 2000.
[5]
C. Parisot, M. Antonini, M. Barlaud, C. Lambert-Nebout, C. Latry and G. Moury, "On Board Strip-based Wavelet Image Coding for Future Space Remote Sensing Missions”, in IGARSS conference, Hawai, USA, 2000.
[6]
B. Usevitch, "Optimal Bit allocation for Biorthogonal Wavelet Coding", in Proc. Data Compression Conference (Snowbird, Utah), pp. 387-395, Mars 1996.
[7]
P. Charbonnier, L. Blanc-Féraud, G. Aubert and M. Barlaud, "Deterministic Edge-Preserving Regularization in Computed Imaging", IEEE Transaction on Image Processing, vol. 5, n µ 12, 1997.
[8]
S. Tramini, M. Antonini and M. Barlaud, "Intraframe Image Decoding based on a Nonlinear Variational Approach", International Journal of Imaging Systems and Technology, vol. 9, pp. 369-380, 1998.
[9]
S. Tramini, M. Antonini, M. Barlaud, G. Aubert, B. Rougé and C. Latry, "Spatio-frequency noise distribution a priori for satellite image joint denoising/deblurring", in ICIP, Vancouver, Canada, 2000.
[10]
S. Tramini, M. Antonini, M. Barlaud and G. Aubert, "Quantization Noise Removal for Optimal Transform Decoding", ICIP, vol. 3, Chicago, Illinois, USA, Oct. 1998.
[11]
J. Kalifa, S. Mallat, F. Falzon and B. Rougé, "High resolution satellite image restoration with frames", in SPIE Conference on Wavelets Applications in Signal and Image Proscessing IV, Denver, Colorado, USA, 1996.
7. REFERENCES [1]
« « °¯ « u¬®
[2]
Table 1. PSNR results on Nîmes. ing/decoding structure takes account of information provided by the acquisition system (MTF, acquisition noise characteristics and sampling scheme) and relationships between coding and decoding in order to improve the performances of the whole image processing chain. The results presented in this paper illustrate the behaviour of our proposed work; they show an appreciable improvement in visual quality, since noise artifacts are removed and high frequency details are restored. This coding/decoding structure can be applied to any optical acquisition system i.e. which blurs and/or adds acquisition noise and/or uses compression.
w« u « u t¤£¥§¦¨3©Mª « « u¬ « « « « «
