Bayesian sparsity enforcing methods for general inverse problems Ali Mohammad-Djafari Laboratoire des Signaux et Syst`emes, UMR8506 CNRS-SUPELEC-UNIV PARIS SUD 11 SUPELEC, 91192 Gif-sur-Yvette, France http://lss.supelec.free.fr Email: [email protected] http://djafari.free.fr Seminar at Sharif University, EECS Dep. December 16, 2014 A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 1/71

Contents Sparse signals and images First ideas for using sparsity in signal processing Modeling for sparse representation Bayesian Maximum A Posteriori (MAP) approach and link with Deterministic Regularization 5. Priors which enforce sparsity 1. 2. 3. 4.

◮ ◮ ◮ ◮

Heavy tailed: Double Exponential, Generalized Gaussian, ... Mixture models: Mixture of Gaussians, Student-t, ... Hierarchical models with hidden variables General Gauss-Markov-Potts models

6. Computational tools: Joint Maximum A Posteriori (JMAP), MCMC and Variational Bayesian Approximation (VBA) 7. Applications in Inverse Problems: X ray Computed Tomography, Microwave and Ultrasound imaging, Sattelite and Hyperspectral image processing, Spectrometry, CMB, ... A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 2/71

1. Sparse signals and images ◮

Sparse signals: Direct sparsity 3

4

3.5 2 3 1 2.5

0

2

1.5 −1 1 −2 0.5

−3

◮

0

20

40

60

80

100

120

140

160

180

0

0

20

40

60

80

100

120

140

160

180

Sparse images: Direct sparsity 50

100

150

200

250 50

100

150

200

250

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 3/71

Sparse signals and images ◮

Sparse signals in a Transform domain 1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

−0.2

−0.2

−0.4

−0.4

−0.6

−0.6

−0.8

−1

◮

−0.8

0

20

40

60

80

100

120

140

160

180

200

−1

0

20

40

60

80

100

120

140

160

180

200

Sparse images in a Transform domain

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 4/71

Sparse signals and images ◮

Sparse signals in Fourier domain Time domain

Fourier domain

6

200

180 4 160

140 2 120

0

100

80 −2 60

40 −4 20

−6

◮

0

20

40

60

80

100

120

140

160

180

200

Sparse images in wavelet domain Space domain

0

0

50

100

150

200

250

Fourier domain

50

100

150

200

250 50

100

150

200

250

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 5/71

Sparse signals and images ◮

Sparse signals: Sparsity in a Transform domaine

3

6

2.5

4

2

2

1.5

0

1

−2

0.5

−4

1

0.8

0.6

0.4

0.2

0

−0.2

−0.4

−0.6

−0.8

0

0

20

40

60

80

100

120

140

160

180

200

3

−6

0

20

40

60

80

100

120

140

160

180

−1

200

0

20

40

60

80

100

120

140

160

180

200

180

200

200

1 180 2

2

160

140

3

1 120

4 0

100

5 80 −1

6

60

40

7

−2 20

8 −3

0

20

40

60

80

100

120

140

160

180

200

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20

40

60

80

100

120

140

160

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 6/71

Sparse signals and images

50

50

100

100

150

150

200

200

250

250 50

100

150

200

250

50

100

150

200

250

50

100

150

200

250

50

100

150

200

250

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 7/71

Sparse signals and images (Fourier and Wavelets domain)

Image

Fourier

3500

Wavelets

6000

10 3000

5000

8

2500 4000 2000

6 3000

1500

4 2000 1000

0

2

1000

500

0

50

100

150

200

0

250

0

Image hist.

2

4

6

8

10

12

14

0 16 −500

Fourier coeff. hist. 1400

1400

1200

1200

1200

1000

1000

1000

800

800

800

600

600

600

400

400

400

200

200

−150

−100

−50

0

50

bands 1-3

100

150

200

0 −200

500

Wavelet coeff. hist.

1400

0 −200

0

200

−150

−100

−50

0

50

bands 4-6

100

150

200

0 −200

−150

−100

−50

0

50

bands 7-9

100

150

200

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 8/71

2. First ideas: some history ◮

◮

1948: Shannon: Sampling theorem and reconstruction of a band limited signal 1993-2007: ◮

◮ ◮

◮

◮

Mallat, Zhang, Cand`es, Romberg, Tao and Baraniuk: Non linear sampling, Compression and reconstruction, Fuch: Sparse representation Donoho, Elad, Tibshirani, Tropp, Duarte, Laska: Compressive Sampling, Compressive Sensing

2007-2012: Algorithms for sparse representation and compressive Sampling: Matching Pursuit (MP), Projection Pursuit Regression, Pure Greedy Algorithm, OMP, Basis Poursuit (BP), Dantzig Selector (DS), Least Absolute Shrinkage and Selection Operator (LASSO), Iterative Hard Thresholding... 2003-2012: Bayesian approach to sparse modeling Tipping, Bishop: Sparse Bayesian Learning, Relevance Vector Machine (RVM), ...

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 9/71

3. Modeling and representation Modeling via a basis (codebook, overcomplete dictionnary, Design Matrix)

◮

g(t) =

N X

f j φj (t), t = 1, · · · , T −→ g = Φ′ f

j=1

g(t)

φj (t)

gi

fj

phij(i)

fj

2

1

1

1.5

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

1

0.5

0 0

0

−0.2

−0.2

−0.4

−0.4

−0.6

−0.6

−0.5

−1

−1.5

−2

−2.5

−0.8

0

10

20

30

40

50

60

T = 100

70

80

90

100

−1

−0.8

0

10

20

30

40

50

60

70

80

90

100

−1

10

15

20

25

30

35

N = 35

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 10/71

3. Modeling and representation ◮

Modeling via a basis (codebook, overcomplete dictionnary, Design Matrix) g(t) =

N X

f j φj (t), t = 1, · · · , T −→ g = Φ′ f

j=1

◮

When T ≥ N

2 N T X X b f j = arg min g(t) − f j φj (t) −→ fj t=1 j=1 b = arg min kg − Φ′ f k2 = [ΦΦ′ ]−1 Φg f f

◮

◮

b = Φg When orthogonal basis: ΦΦ′ = I −→ f fbj =

N X

g(t) φj (t) =< g(t), φj (t) >

t=1

Application in Compression, Transmission and Decompression

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 11/71

Modeling and representation ◮

When overcomplete basis N > T : Infinite number of solutions for Φ′ f = g. We have to select one: b = arg min f kf k22 ′ f : Φ f =g

or writing differently:

minimize kf k22 subject to Φ′ f = g resulting to: ◮ ◮ ◮

b = Φ[Φ′ Φ]−1 g f Again if Φ′ Φ = I −→ fb = Φg. No real interest if we have to keep all the N coefficients: Sparsity: minimize kf k0 subject to Φ′ f = g or minimize kf k1 subject to Φ′ f = g

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 12/71

Sparse decomposition ◮

Strict sparsity and exact reconstruction minimize kf k0 subject to Φ′ f = g kf k0 is the number of non-zero elements of f ◮ ◮ ◮ ◮ ◮

◮

Matching Pursuit (MP) [Mallat & Zhang, 1993] Orthogonal Matching Pursuit (OMP) [Lin, Huang et al., 1993] Projection Pursuit Regression Greedy Algorithms Iterative Hard Thresholding (IHT) [Marvasti et al]

Sparsity enforcing and exact reconstruction minimize kf k1 subject to Φ′ f = g ◮ ◮

Basis Pursuit (BP) Block Coordinate Relaxation

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 13/71

Sparse decomposition ◮

Strict sparsity and approximate reconstruction

minimize kf k0 subject to kg − Φ′ f k2 < c b = arg min kf k0 + µkg − Φ′ f k2 = arg min kg − Φ′ f k2 + λkf k0 f f

◮

f

Sparsity enforcing and approximate reconstruction b = arg min kg − Φ′ f k2 + λkf k1 f f

J(f ) = kg − Φ′ f k2 + λkf k1 = kg − Φ′ f k2 + λ

X

|f j |

j

◮

Main Algorithm: LASSO [Tibshirani 2003] minimize kg − Φ′ f k2 subject to kf k1 < τ

A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 14/71

Sparse Decomposition Algorithms ◮

LASSO: J(f ) = kg − Φ′ f k2 + λ

X

|f j |

j

◮

Other Criteria ◮

Lp J(f ) = kg − Φ′ f k2 + λ1

X

|f j |p ,

1