.
Bayesian sparsity enforcing methods for general inverse problems Ali Mohammad-Djafari Laboratoire des Signaux et Syst`emes, UMR8506 CNRS-SUPELEC-UNIV PARIS SUD 11 SUPELEC, 91192 Gif-sur-Yvette, France http://lss.supelec.free.fr Email:
[email protected] http://djafari.free.fr Seminar at Sharif University, EECS Dep. December 16, 2014 A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 1/71
Contents Sparse signals and images First ideas for using sparsity in signal processing Modeling for sparse representation Bayesian Maximum A Posteriori (MAP) approach and link with Deterministic Regularization 5. Priors which enforce sparsity 1. 2. 3. 4.
◮ ◮ ◮ ◮
Heavy tailed: Double Exponential, Generalized Gaussian, ... Mixture models: Mixture of Gaussians, Student-t, ... Hierarchical models with hidden variables General Gauss-Markov-Potts models
6. Computational tools: Joint Maximum A Posteriori (JMAP), MCMC and Variational Bayesian Approximation (VBA) 7. Applications in Inverse Problems: X ray Computed Tomography, Microwave and Ultrasound imaging, Sattelite and Hyperspectral image processing, Spectrometry, CMB, ... A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 2/71
1. Sparse signals and images ◮
Sparse signals: Direct sparsity 3
4
3.5 2 3 1 2.5
0
2
1.5 −1 1 −2 0.5
−3
◮
0
20
40
60
80
100
120
140
160
180
0
0
20
40
60
80
100
120
140
160
180
Sparse images: Direct sparsity 50
100
150
200
250 50
100
150
200
250
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 3/71
Sparse signals and images ◮
Sparse signals in a Transform domain 1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
−0.2
−0.2
−0.4
−0.4
−0.6
−0.6
−0.8
−1
◮
−0.8
0
20
40
60
80
100
120
140
160
180
200
−1
0
20
40
60
80
100
120
140
160
180
200
Sparse images in a Transform domain
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 4/71
Sparse signals and images ◮
Sparse signals in Fourier domain Time domain
Fourier domain
6
200
180 4 160
140 2 120
0
100
80 −2 60
40 −4 20
−6
◮
0
20
40
60
80
100
120
140
160
180
200
Sparse images in wavelet domain Space domain
0
0
50
100
150
200
250
Fourier domain
50
100
150
200
250 50
100
150
200
250
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 5/71
Sparse signals and images ◮
Sparse signals: Sparsity in a Transform domaine
3
6
2.5
4
2
2
1.5
0
1
−2
0.5
−4
1
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
0
0
20
40
60
80
100
120
140
160
180
200
3
−6
0
20
40
60
80
100
120
140
160
180
−1
200
0
20
40
60
80
100
120
140
160
180
200
180
200
200
1 180 2
2
160
140
3
1 120
4 0
100
5 80 −1
6
60
40
7
−2 20
8 −3
0
20
40
60
80
100
120
140
160
180
200
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
20
40
60
80
100
120
140
160
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 6/71
Sparse signals and images
50
50
100
100
150
150
200
200
250
250 50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 7/71
Sparse signals and images (Fourier and Wavelets domain)
Image
Fourier
3500
Wavelets
6000
10 3000
5000
8
2500 4000 2000
6 3000
1500
4 2000 1000
0
2
1000
500
0
50
100
150
200
0
250
0
Image hist.
2
4
6
8
10
12
14
0 16 −500
Fourier coeff. hist. 1400
1400
1200
1200
1200
1000
1000
1000
800
800
800
600
600
600
400
400
400
200
200
−150
−100
−50
0
50
bands 1-3
100
150
200
0 −200
500
Wavelet coeff. hist.
1400
0 −200
0
200
−150
−100
−50
0
50
bands 4-6
100
150
200
0 −200
−150
−100
−50
0
50
bands 7-9
100
150
200
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 8/71
2. First ideas: some history ◮
◮
1948: Shannon: Sampling theorem and reconstruction of a band limited signal 1993-2007: ◮
◮ ◮
◮
◮
Mallat, Zhang, Cand`es, Romberg, Tao and Baraniuk: Non linear sampling, Compression and reconstruction, Fuch: Sparse representation Donoho, Elad, Tibshirani, Tropp, Duarte, Laska: Compressive Sampling, Compressive Sensing
2007-2012: Algorithms for sparse representation and compressive Sampling: Matching Pursuit (MP), Projection Pursuit Regression, Pure Greedy Algorithm, OMP, Basis Poursuit (BP), Dantzig Selector (DS), Least Absolute Shrinkage and Selection Operator (LASSO), Iterative Hard Thresholding... 2003-2012: Bayesian approach to sparse modeling Tipping, Bishop: Sparse Bayesian Learning, Relevance Vector Machine (RVM), ...
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 9/71
3. Modeling and representation Modeling via a basis (codebook, overcomplete dictionnary, Design Matrix)
◮
g(t) =
N X
f j φj (t), t = 1, · · · , T −→ g = Φ′ f
j=1
g(t)
φj (t)
gi
fj
phij(i)
fj
2
1
1
1.5
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
1
0.5
0 0
0
−0.2
−0.2
−0.4
−0.4
−0.6
−0.6
−0.5
−1
−1.5
−2
−2.5
−0.8
0
10
20
30
40
50
60
T = 100
70
80
90
100
−1
−0.8
0
10
20
30
40
50
60
70
80
90
100
−1
10
15
20
25
30
35
N = 35
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 10/71
3. Modeling and representation ◮
Modeling via a basis (codebook, overcomplete dictionnary, Design Matrix) g(t) =
N X
f j φj (t), t = 1, · · · , T −→ g = Φ′ f
j=1
◮
When T ≥ N
2 N T X X b f j = arg min g(t) − f j φj (t) −→ fj t=1 j=1 b = arg min kg − Φ′ f k2 = [ΦΦ′ ]−1 Φg f f
◮
◮
b = Φg When orthogonal basis: ΦΦ′ = I −→ f fbj =
N X
g(t) φj (t) =< g(t), φj (t) >
t=1
Application in Compression, Transmission and Decompression
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 11/71
Modeling and representation ◮
When overcomplete basis N > T : Infinite number of solutions for Φ′ f = g. We have to select one: b = arg min f kf k22 ′ f : Φ f =g
or writing differently:
minimize kf k22 subject to Φ′ f = g resulting to: ◮ ◮ ◮
b = Φ[Φ′ Φ]−1 g f Again if Φ′ Φ = I −→ fb = Φg. No real interest if we have to keep all the N coefficients: Sparsity: minimize kf k0 subject to Φ′ f = g or minimize kf k1 subject to Φ′ f = g
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 12/71
Sparse decomposition ◮
Strict sparsity and exact reconstruction minimize kf k0 subject to Φ′ f = g kf k0 is the number of non-zero elements of f ◮ ◮ ◮ ◮ ◮
◮
Matching Pursuit (MP) [Mallat & Zhang, 1993] Orthogonal Matching Pursuit (OMP) [Lin, Huang et al., 1993] Projection Pursuit Regression Greedy Algorithms Iterative Hard Thresholding (IHT) [Marvasti et al]
Sparsity enforcing and exact reconstruction minimize kf k1 subject to Φ′ f = g ◮ ◮
Basis Pursuit (BP) Block Coordinate Relaxation
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 13/71
Sparse decomposition ◮
Strict sparsity and approximate reconstruction
minimize kf k0 subject to kg − Φ′ f k2 < c b = arg min kf k0 + µkg − Φ′ f k2 = arg min kg − Φ′ f k2 + λkf k0 f f
◮
f
Sparsity enforcing and approximate reconstruction b = arg min kg − Φ′ f k2 + λkf k1 f f
J(f ) = kg − Φ′ f k2 + λkf k1 = kg − Φ′ f k2 + λ
X
|f j |
j
◮
Main Algorithm: LASSO [Tibshirani 2003] minimize kg − Φ′ f k2 subject to kf k1 < τ
A. Mohammad-Djafari, Bayesian sparsity enforcing methods for general inverse problems, Sharif University, December 16, 2014, 14/71
Sparse Decomposition Algorithms ◮
LASSO: J(f ) = kg − Φ′ f k2 + λ
X
|f j |
j
◮
Other Criteria ◮
Lp J(f ) = kg − Φ′ f k2 + λ1
X
|f j |p ,
1
