Fast differential operators for real-time edge detection HDU Workshop on smarthome technologies and applications
C. Dinh Nguyen, M. Delalandre, D. Conte, T.A. Pham LIFAT Laboratory, RFAI group, Tours city, France
[email protected]
November 4, 2017
1 / 32
Mathieu Delalandre - CV in short I I I I I I
PhD in computer science with 12 years of experience, Assistant Professor at the LIFAT Lab (Tours city, France), image processing (local detectors, template matching), application domains (document and video analysis), training / projects (PhD supervisor of Cong Nguyen, startup ImageStream, ScannerLoire project), more about myself: http://mathieu.delalandre.free.fr/
2 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
3 / 32
Introduction Edge detection is a fundamental image processing task that serves for key-point detection, segmentation, features extraction, object detection, etc. [N. Petkov, 2015].
Real-time systems respect the correctness of the results and a deadline (e.g. 10-60 FPS / 1-10 ms / 1-8 Mp). Differential operators are the most suitable for real-time [S. Navale, 2014] compared to state-of-the-art (statistical, active contours, multi-resolution, etc.) [N. Petkov, 2015]. 4 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
5 / 32
Mathematical definitions of derivative operators (1/2)
Definition: g (x, y ) is a mathematical function to be applied for averaging, first and second order derivatives are applied to get the gradient ∇g and the Laplacian ∇2 g Eq. (1). " ∂g # g ∂x ∇2 g = gxx + gyy (1) ∇g = x = ∂g gy ∂y Responses of the operators are controlled with magnitude and orientation features mag (∇g ), α(∇f ) and hysteresis thresholding for the gradient [D. Sen, 2010], zero-crossing for the Laplacian.
6 / 32
Mathematical definitions of derivative operators (2/2) Operators can be generic employing gaussian, box filter, exponential, sinusoid, etc. functions g (x, y ), or designed with training [S. Konishi, 2003] so infinite . . . , e.g. g (x, y ) is a gaussian.
7 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
8 / 32
Edge detection with convolution (1/4) Principle: g (x, y ) is discretized and normalized to get a discrete mask of size ω 2 .
Convolution is obtained with application of Eq. (2) considering f (x, y ) as a discrete image.
h(x, y ) = f (x, y ) ⊗ g (x, y ) =
a b X X
g (s, t)f (x + s, y + t) (2)
s=−a t=−b
h = f ⊗ g is referred as the general product to the whole image. 9 / 32
Edge detection with convolution (2/4) Complexity: for an image of size N, f ⊗ g requires O(Nω 2 ) operations. Optimization can be obtained with (i) LUT [S. Liqin, 1994], (ii) parallelization [L.J. Siegel, 1982] [G. Mitra, 2013] and (iii) separability Eq. (3) that shifts the complexity to O(Nω). f ⊗ g = f ⊗ (g1 ⊗ g2 ) = (f ⊗ g1 ) ⊗ g2 e.g. g1 ⊗ g2 = g 1 −1 0 1 2 ⊗ −1 0 1 = −2 0 2 1 −1 0 1
(3)
(4)
10 / 32
Edge detection with convolution (3/4) Reformulation: not all the operators can be implemented with separability, some require a mathematical reformulation e.g. if g (x, y ) is a gaussian function, ∇2 g (x, y ) ' g1 (x, y ) − g2 (x, y ), this is the LoG reformulation into a DoG operator.
2
2
− x +y2 x 2 + y 2 − 2σ 2 2σ exp ∇2 g (x, y ) = (5) σ4 ' g1 (x, y ) − g2 (x, y ) # # " " 2 2 2 2 1 − x 2σ+y2 1 1 − x 2σ+y2 1 1 2 e −√ e ' √ 2π σ1 2π σ2
LoG
' DoG
11 / 32
Edge detection with convolution (4/4) Scale: detection results depend of the ω parameter controlled with scale selection [C. Lopez-Molina, 2013]. When ω is large, processing time at O(Nω) is not compatible with real-time.
2.2 GHz CPU - OpenCV / C++ - single thread 12 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
13 / 32
Edge detection with image integral (1/6) The image integral is double discrete sum Eq.(6). s(x, y ) =
XX
f (i, j)
(6)
i≤x j≤y
With a proper reformulation of s(x, y ), the integral of the whole image requires O(2N) operations [P. Viola, 2004].
Integral image with a box filter [7 × 7] 14 / 32
Edge detection with image integral (2/6) The correlation of an averaging product requires 4 accesses, the comparison of two adjacent boxes 6 accesses. a, b, c, d, e and f are some s(x, y ) values, we have
A1 = (a + c) − (b + d) = (a − d) + (c − b)
(7)
A2 = (b + f ) − (c + e) = (f − e) + (b − c) A2 − A1 = (f − e) + 2(b − c) + (d − a)
Thus, complexity is independent of the ω parameter at O(N). 15 / 32
Edge detection with image integral (3/6) The A2 − A1 difference results in a correlation product h = f ⊗ gx with a non-symmetric boxcar operator [C.F. Carames, 2014]. This is a haar feature, e.g. −1 −1 1 1 −1 −1 1 1 (8) gx = −1 −1 1 1 −1 −1 1 1
16 / 32
Edge detection with image integral (4/6) Q The repeated averaging with a box filter set i (x, y ), with i ∈ [1, n] the sizes, approximates a Gaussian filter. It is referred as the almost-gaussian operator [E. Elboher, 2012] [P. Kovesi, 2010]. g (x, y , σ) ' gb(x, y , σ) =
X Y (x, y )
(9)
i∈[1,n] i
We can set n ∈ [4, 6] for a good approximation [P. Kovesi, 2010].
17 / 32
Edge detection with image integral (5/6) The averaging product with a box filter of size i can be obtained from image integral. Ai = (a − d) + (c − b) = f (x, y ) ⊗
Q
i (x, y )
(10)
With the linearity property of convolution, we can formulate a difference of almost-gaussian i.e. a CABOX operator [V. Fragoso, 2014]. This is a DoG estimator LoG ' DoG ' CABOX . X Y
(x, y ) ⊗ f (x, y ) −
i∈[1,n] i
X Y
(x, y ) ⊗ f (x, y ) (11)
j∈[1,n] j
X Y X Y = (x, y ) − (x, y ) ⊗ f (x, y ) i∈[1,n] i
j∈[1,n] j
= (b g (x, y , σ1 ) − gb(x, y , σ2 )) ⊗ f (x, y ) 18 / 32
Edge detection with image integral (6/6)
As an average, detection with image integral offers intermediate accelerations little compatible with real-time constraints.
2.2 GHz CPU - OpenCV / C++ - single thread
19 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
20 / 32
Edge detection with sublinear methods (O(N β∈(0,1] )) (1/3)
The sublinear algorithms can accelerate the detection process at a O(N β∈(0,1] ) complexity, e.g. [I. Horev, 2015].
21 / 32
Edge detection with sublinear methods (O(N β∈(0,1] )) (2/3)
Gridding divides the image into patches of size n × n. Sampling: a discrete array a = {a1 , ...., as } is extracted from each 2 patch respecting a sufficient sampling with s = n4 . Statistical test: a detection rule is applied to each patch combining two probability density functions Pµ0 , Pµ1 .
f (µ1 , µ2 , a) = Pµ0 log Pµi
=
Pµ0 Pµ1
(12)
1 1 exp( K ||a−µi ||) K
22 / 32
Edge detection with sublinear methods (O(N β∈(0,1] )) (3/3)
Statistical test: with gridding we have a set of ai,j vectors where f (µ1 , µ2 , ai,j ) takes part of a 2D Kullback–Leibler divergence. DKL (Pµ0 ||Pµ1 ) =
XX ∀i
f (µ1 , µ2 , ai,j )
(13)
∀j
The minimization of f (µ1 , µ2 , ai,j ) will target the detection of noisy regions at intensity µ0 with clear visible power lines at intensity µ1 . Edge detection: after testing, C patches are processed with boxcar operators at padding areas, then line tracing algorithm. Complexity is mainly dependent of the sampling step at O(N/4).
23 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
24 / 32
Comparison of techniques (1/1) As a general trend, the optimization of methods raises problems of parameter control and non exhaustivity of the detection [I.Horev, 2015] [V.Fragoso, 2014].
E/NE means exhaustive/non exhaustive methods, β ∈ (0, 1], Q P (*) means a box set i∈[1,n] i (x, y ) for a σi approximation.
25 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
26 / 32
Work in progress (1/2) Edge detection with image integral interpolation: local image integral can be interpolated at O(N/∆2 ). When the sampling parameter is average, the interpolation error is low.
27 / 32
Work in progress (2/2) Edge detection with image integral interpolation: such an approach raises different open problems. I I I
the relation between ω, ∆ need to be defined to get a parameter-free model ∆ = f (ω). the algorithm for interpolation has to be fixed for optimization and noise minimization. the edge ridge should be interpolated from sampled integral to track the operator response.
28 / 32
Summary
CV in short Fast differential operators for real-time edge detection Introduction Mathematical formulation of operators Edge detection with convolution O(Nω) Edge detection with image integral O(N) Edge detection with sublinear methods (O(N β∈(0,1] )) Comparison of techniques Work in progress Questions
29 / 32
Questions Thank for your attention, any questions . . . Fast differential operators for real-time edge detection C. Dinh Nguyen, M. Delalandre, D. Conte, T.A. Pham
30 / 32
References I I
N. Petkov, G. Papari. Edge and line oriented contour detection: State of the art. Image and Vision Computing (IVC), 29(2):79–103, 2011.
I
S. Navale, P.P. Gundewar. Real Time Edge Detection Using DSP Processor. International Journal of Computer Science and Network (IJCSN-), vol 3(6):472-477, 2014.
I
D. Sen and S.K. Pal. Gradient histogram: Thresholding in a region of interest for edge detection. Image and Vision Computing (IVC), 28: 677–695, 2010.
I
S. Konishi and al. Statistical edge detection: learning and evaluating edge cues. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 25(1):57-74, 2003.
I
S. Liqin, and al. Edge detection on real time using LOG filter. International Symposium on Speech, Image Processing and Neural Networks, 1994.
I
L. Siegel and al. A. Parallel Processing Approaches to Image Correlation. Transactions on Computers, C-31, pp. 208-218, 1982.
I
G. Mitra and al. Use of SIMD Vector Operations to Accelerate Application Code Performance on Low-Powered ARM and Intel Platforms. International Symposium on Parallel Distributed Processing Workshops (IPDPSW), pp. 1107-1116, 2013.
I
C. Lopez-Molina and al. Multiscale edge detection based on Gaussian smoothing and edge tracking. Knowledge-Based Systems (KBS), (44):101–111, 2013.
I
P. Viola, P and M.J. Jones. Robust real-time face detection. International Journal of Computer Vision (IJCV), 57(2): 137–154, 2004.
I
C. Fernandez-Carames and al. A Real-time Door Detection System for Domestic Robotic Navigation. Journal of Intelligent and Robotic Systems, (76):119–136, 2014.
I
V. Fragoso and al. Cascade of Box (CABOX) Filters for Optimal Scale Space Approximation. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 126-131, 2014.
I
E. Elboher and M. Werman. Efficient and Accurate Gaussian Image Filtering Using Running Sums. International Conference on Intelligent Systems Design and Applications (ISDA), pp. 897-902, 2012. 31 / 32
References II I
P. Kovesi. Fast Almost-Gaussian Filtering. International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 121-125, 2010.
I
I. Horev and al. Detection of Long Edges on a Computational Budget: A Sublinear Approach. SIAM Journal on Imaging Sciences, 8(1): 458–483, 2015.
32 / 32
