Variational Bayesian Approximation methods for inverse problems Ali Mohammad-Djafari Laboratoire des Signaux et Syst`emes, UMR8506 CNRS-SUPELEC-UNIV PARIS SUD 11 SUPELEC, 91192 Gif-sur-Yvette, France http://lss.supelec.free.fr Email: [email protected] http://djafari.free.fr

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 1/17

1. General inverse problem g(t) = Hf (t) + ǫ(t), g(r) = Hf (r) + ǫ(r),

t ∈ [1, · · · , T ] r = (x, y) ∈ R2

◮

f unknown quantity (input)

◮

H Forward operator: (Convolution, Radon, Fourier or any Linear operator)

◮

g observed quantity (output)

◮

ǫ represents the errors of modeling and measurement

Discretization: g = Hf + ǫ

◮

Forward operation Hf

◮

Adjoint operation H ′ g :

◮

Inverse operation (if exists) H −1 g

A. Mohammad-Djafari,

NCMIP2012,

< H ′ g, f >=< Hf , g >

May 15, 2012, ENS Cachan, France, 2/17

2. General Bayesian Inference ◮

Bayesian inference: p(f |g, θ) =

◮

◮ ◮

p(g|f , θ 1 ) p(f |θ 2 ) p(g|θ)

with θ = (θ 1 , θ 2 ) Point estimators: b Maximum A Posteriori (MAP) or Posterior Mean (PM) −→ f Full Bayesian inference: Simple prior models: p(f |θ 2 )

q(f , θ|g) ∝ p(g|f , θ 1 ) p(f |θ 2 ) p(θ) ◮

Prior models with hidden variables: p(f |z, θ 2 ) p(z|θ 3 ) q(f , z, θ|g) ∝ p(g|f , θ 1 ) p(f |θ 2 ) p(z|θ 3 ) p(θ)

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 3/17

3. Sparsity enforcing prior models ◮

Simple heavy tailed models: ◮ ◮ ◮

◮ ◮

◮

Generalized Gaussian, Double Exponential Student-t, Cauchy Elastic net Symmetric Weibull, Symmetric Rayleigh Generalized hyperbolic

Hierarchical mixture models: ◮ ◮

◮ ◮ ◮ ◮

A. Mohammad-Djafari,

Mixture of Gaussians Bernoulli-Gaussian Mixture of Gammas Bernoulli-Gamma Mixture of Dirichlet Bernoulli-Multinomial

NCMIP2012,

May 15, 2012, ENS Cachan, France, 4/17

4. Simple heavy tailed models • Generalized Gaussian, Double Exponential Y X p(f |γ, β) = GG(fj |γ, β) ∝ exp −γ |fj |β j

j

β = 1 Double exponential or Laplace. 0 < β ≤ 1 are of great interest for sparsity enforcing. • Student-t and Cauchy models p(f |ν) =

Y j

ν+1X 2 St(fj |ν) ∝ exp − log 1 + fj /ν 2 j

Cauchy model is obtained when ν = 1. • Elastic net prior model p(f |ν) = A. Mohammad-Djafari,

Y

j NCMIP2012,

X 2 EN (fj |ν) ∝ exp − (γ1 |fj | + γ2 fj ) May 15, 2012, ENS Cachan, France, j 5/17

5 Mixture models • Mixture of two Gaussians (MoG2) model Y p(f |λ, v1 , v0 ) = (λN (fj |0, v1 ) + (1 − λ)N (fj |0, v0 )) j

• Bernoulli-Gaussian (BG) model Y Y p(f |λ, v) = p(fj ) = (λN (fj |0, v) + (1 − λ)δ(fj )) j

j

• Mixture of Gammas Y p(f |λ, v1 , v0 ) = (λG(fj |α1 , β1 ) + (1 − λ)G(fj |α2 , β2 )) j

• Bernoulli-Gamma model Y p(f |λ, α, β) = [λG(fj |α, β) + (1 − λ)δ(fj )] j

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 6/17

6. MAP, Joint MAP ◮ ◮

Inverse problems: Posterior law:

g = Hf + ǫ

p(f |θ, g) ∝ p(g|f , θ 1 ) p(f |θ 2 ) ◮

◮

Examples: Gaussian noise, Gaussian prior and MAP: b = arg min {J(f )} with J(f ) = kg − Hf k2 + λkf k2 f 2 2 f

Gaussian noise, Double Exponential prior and MAP: b = arg min {J(f )} with J(f ) = kg − Hf k2 + λkf k1 f 2 f Full Bayesian: Joint Posterior:

p(f , θ|g) ∝ p(g|f , θ 1 ) p(f |θ 2 ) p(θ) ◮

Joint MAP:

A. Mohammad-Djafari,

NCMIP2012,

b = arg max {p(f , θ|g)} b , θ) (f (f ,θ )

May 15, 2012, ENS Cachan, France, 7/17

7. Marginal MAP and PM estimates ◮

◮

◮

◮

b = arg max {p(θ|g)} where Marginal MAP: θ θ Z Z p(θ|g) = p(f , θ|g) df = p(g|f , θ 1 ) p(f |θ 2 ) df

n o b g) b = arg max p(f | θ, and then f f Z b= b g) df Posterior Mean: f f p(f |θ, EM and GEM Algorithms

Variational Bayesian Approximation: Approximate p(f , θ|g) by q(f , θ|g) = q1 (f |g) q2 (θ|g) and then continue computations.

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 8/17

8. Hierarchical models and hidden variables ◮

All the mixture models and some of simple models can be modeled via hidden variables z.

◮

Example 1: Student-t model

( ◮

◮

p(f |z) =

Q

j p(fj |zj ) =

Q

o n 1P 2 z f N (f |0, 1/z ) ∝ exp − j j j j j j 2

(a−1)

p(zj |a, b) = G(zj |a, b) ∝ zj

exp {−bzj } with a = b = ν/2

Example 2: MoG model: p(f |z) =

Q

j p(fj |zj ) =

P (zj = 1) = λ,

Q

P ∝ exp − 12 j N f |0, v j zj j

P (zj = 0) = 1 − λ

With these models we have: p(f , z, θ|g) ∝ p(g|f , θ 1 ) p(f |z, θ 2 ) p(z|θ 3 ) p(θ)

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 9/17

fj2 vzj

9. Bayesian Computation and Algorithms ◮

Often, the expression of p(f , z, θ|g) is complex.

◮

Its optimization (for Joint MAP) or its marginalization or integration (for Marginal MAP or PM) is not easy

◮

Two main techniques: MCMC and Variational Bayesian Approximation (VBA)

◮

MCMC: Needs the expressions of the conditionals p(f |z, θ, g), p(z|f , θ, g), and p(θ|f , z, g)

◮

VBA: Approximate p(f , z, θ|g) by a separable one q(f , z, θ|g) = q1 (f ) q2 (f ) q3 (θ) and do any computations with these separable ones.

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 10/17

10. Bayesian Variational Approximation ◮

◮

◮

Objective: Approximate p(f , z, θ|g) by a separable one q(f , z, θ|g) = q1 (f ) q2 (f ) q3 (θ) Criterion: Z q q KL(q : p) = q ln = ln p p q Free energy: KL(q : p) = ln p(g|M) − F(q) where: Z Z Z p(g|M) = p(f , z, θ, g|M) df dz dθ

with p(f , z, θ, g|M) = p(g|f , θ) p(f |z, θ) p(z|θ) p(θ) and F(q) is the free energy associated to q defined as p(f , z, θ, g|M) F(q) = ln q(f , z, θ) q ◮

For a given model M, minimizing KL(q : p) is equivalent to maximizing F(q) and when optimized, F(q ∗ ) gives a lower bound for ln p(g|M).

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 11/17

11. BVA with Student-t priors Scale Mixture Model of Student-t: Z ∞ St(f j |ν) = N (f j |0, 1/τ j ) G(τ j |ν/2, ν/2) dτ j 0

Hidden variables τ j : p(f |τ )

=

Q

j p(f j |τ j ) =

Q

n o 1P 2 N (f |0, 1/τ ) ∝ exp − τ f j j j j j j 2

(α−1)

p(τ j |α, β) = G(τ j |α, β) ∝ τ j

exp {−βτ j } with α = β = ν/2

Cauchy model is obtained when ν = 1: ◮

Graphical model: - f Hn ? R @ - g αǫ0 , βǫ0 - τn ǫ- ǫ

ατ 0 , βτ 0- τn

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 12/17

12. BVA with Student-t priors Algorithm p(g|f , τǫ ) = N (g|Hf , (1/τǫ )I) αj , βej ) q2j (τ j ) = G(τ j |e p(τǫ |ατ 0 , βQ τ 0 ) = G(τǫ |ατ 0 , βτ 0 ) e < f >= µ α e = α + 1/2 j 00 p(f |τ ) = j N (f j |0, 1/τ j ) ′ e +µ 2 eµ e′ < f f >= Σ e Q βj = β00 + < f j > /2 p(τ |α0 , β0 ) = j G(τ j |α0 , β0 ) e jj + µ e2j < f 2j >= [Σ] e e e q (τ ), , β ) = G(τ |e α 3 ǫ q1 (f |e τǫ ǫ τǫ µ, Σ) = N (f |e µ, Σ) e=α λ eτ /βeτ α eτǫ = ατ 0 + (n + 1)/2 e ′g e µ = hλi ΣH βeτǫ = βτ 0 + 1/2[kgk2 Σ e = (hλi H ′ H + Z) e −1 , ej /βej τej = α ′ ′ ′ ′ −2 hfi H g + H hf f i H] e =T e −1 = diag [e with Z τ] e = N (f, e Σ) e τ , λ) e q1 (f |e λ −→

6

e f −→

ej ) e ) = G(τ j |α ej , β q2j (τ j |f

α e j = α00 + n+1 2 D E e =λ e ΣH e ′g f ej = β00 + 1 f 2 e e τ Σ β −→ e j e ′H + T e −1 )−1 −→ 2 Σ = (λH ej τ ej = α e j /β

6

A. Mohammad-Djafari,

NCMIP2012,

eτ ) e ) = G(τ |α eτ , β q3 (τ |f e α e f e τ = ατ 0 + n+1 −→ −λ → 2 e β eτ = βτ 0 + 1 [kgk2 Σ −→ 2 ′ ′ ′ ′ e τ τ ej − 2 < f > H g + H < f f > H] −→ −→ e=α eτ λ e τ /β

May 15, 2012, ENS Cachan, France, 13/17

13. Implementation issues ◮

In inverse problems, often we do not have access directly to the matrix H. But, we can compute: ◮ ◮

◮

◮

Forward operator : Hf −→ g Adjoint operator : H ′ g −→ f

g=direct(f,...) f=transp(g,...)

For any particular application, we can always write two programs (direct & transp) corresponding to the application of these two operators. e , we use a gradient based optimization To compute f algorithm which will use these operators.

◮

We may also need to compute the diagonal elements of [H ′ H].. We also developped algorithms which computes these diagonal elements with the same programs (direct & transp)

A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 14/17

14. Conclusions and Perspectives ◮

◮

◮

◮

◮

We proposed a list of different probabilistic prior models which can be used for sparsity enforcing. We classified these models in two categories: simple heavy tails and hierarchical mixture models We showed how to use these models for inverse problems where the desired solutions are sparse Different algorithms have been developed and their relative performances are compared. We use these models for inverse problems in different signal and image processing applications such as: ◮ ◮ ◮

◮ ◮ ◮

A. Mohammad-Djafari,

Period estimation in biological time series X ray Computed Tomography, Signal deconvolution in Proteomic and molecular imaging Diffraction Optical Tomography Microwave Imaging, Acoustic imaging and sources localization Synthetic Aperture Radar (SAR) Imaging NCMIP2012,

May 15, 2012, ENS Cachan, France, 15/17

15. References 1. A. Mohammad-Djafari, “Bayesian approach with prior models which enforce sparsity in signal and image processing,” EURASIP Journal on Advances in Signal Processing, vol. Special issue on Sparse Signal Processing, (2012). 2. S. Zhu, A. Mohammad-Djafari, H. Wang, B. Deng, X. Li and J. Mao J, “Parameter estimation for SAR micromotion target based on sparse signal representation,” EURASIP Journal on Advances in Signal Processing, vol. Special issue on Sparse Signal Processing, (2012). 3. N. Chu, J. Picheral and A. Mohammad-Djafari, “A robust super-resolution approach with sparsity constraint for near-field wideband acoustic imaging,” IEEE International Symposium on Signal Processing and Information Technology pp 286–289, Bilbao, Spain, Dec14-17,2011 4. N. Bali and A. Mohammad-Djafari, “Bayesian Approach With Hidden Markov Modeling and Mean Field Approximation for Hyperspectral Data Analysis,” IEEE Trans. on Image Processing 17: 2. 217-225 Feb. (2008). 5. J. Griffin and P. Brown, “Inference with normal-gamma prior distributions in regression problems,” Bayesian Analysis, 2010. 6. N. Polson and J. Scott., “Shrink globally, act locally: sparse Bayesian regularization and prediction,” Bayesian Statistics 9, 2010. 7. T. Park and G. Casella., “The Bayesian Lasso,” Journal of the American Statistical Association, 2008. 8. C. F´ evotte and S. Godsill, “A Bayesian aproach for blind separation of sparse source,” IEEE Transactions on Audio, Speech, and Language processing, 2006. 9. H. Snoussi and J. Idier., “Bayesian blind separation of generalized hyperbolic processes in noisy and underdeterminate mixtures,” IEEE Trans. on Signal Processing, 2006. 10. J. R. H. Ishwaran, “Spike and slab variable selection: Frequentist and Bayesian strategies,” Annals of Statistics, 2005. 11. M. Tipping, “Sparse Bayesian learning and the relevance vector machine,” Journal of Machine Learning Research, 2001. A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 16/17

14. References .. 1. Abib. Doucet and P. Duvaut, “Bayesian estimation of state-space models applied to deconvolution of Bernoulli-Gaussian processes,” Signal Processing, vol. 57, no. 2, 1997. 2. P. Williams, “Bayesian regularization and pruning using a Laplace prior,” Neural Computation, 1995. 3. M. Lavielle, “Bayesian deconvolution of Bernoulli-Gaussian processes,” Signal Processing, vol. 33, pp. 67–79, 1993. 4. T. Mitchell and J. Beauchamp, “Bayesian variable selection in linear regression,” Journal of the American Statistical Association, 1988. 5. J. J. Kormylo and J. M. Mendel, “Maximum-likelihood detection and estimation of Bernoulli-Gaussian processes,” vol. 28, pp. 482–488, 1982. 6. H. Snoussi and A. Mohammad-Djafari, “ Estimation of Structured Gaussian Mixtures: The Inverse EM Algorithm,” IEEE Trans. on Signal Processing 55: 7. 3185-3191 July (2007). 7. N. Bali and A. Mohammad-Djafari, “A variational Bayesian Algorithm for BSS Problem with Hidden Gauss-Markov Models for the Sources,” in: Independent Component Analysis and Signal Separation (ICA 2007) Edited by:M.E. Davies, Ch.J. James, S.A. Abdallah, M.D. Plumbley. 137-144 Springer (LNCS 4666) (2007). 8. N. Bali and A. Mohammad-Djafari, “Hierarchical Markovian Models for Joint Classification, Segmentation and Data Reduction of Hyperspectral Images” ESANN 2006, September 4-8, Belgium. (2006) 9. M. Ichir and A. Mohammad-Djafari, “Hidden Markov models for wavelet-based blind source separation,” IEEE Trans. on Image Processing 15: 7. 1887-1899 July (2005) 10. S. Moussaoui, C. Carteret, D. Brie and A Mohammad-Djafari, “Bayesian analysis of spectral mixture data using Markov Chain Monte Carlo methods sampling,” Chemometrics and Intelligent Laboratory Systems 81: 2. 137-148 (2005). 11. H. Snoussi and A. Mohammad-Djafari, “Fast joint separation and segmentation of mixed images” Journal of Electronic Imaging 13: 2. 349-361 April (2004) 12. H. Snoussi and A. Mohammad-Djafari, “Bayesian unsupervised learning for source separation with mixture of Gaussians prior,” Journal of VLSI Signal Processing Systems 37: 2/3. 263-279 June/July (2004) A. Mohammad-Djafari,

NCMIP2012,

May 15, 2012, ENS Cachan, France, 17/17