Real-Time Parallel Implementation of a Particle Filter based Visual Tracking Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-Thierry Laprest´ e Lasmea, Clermont-Ferrand, France Lasmea, UMR 6602 CNRS/Blaise Pascal University Clermont-Ferrand, France
May 12, 2006
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Contents
1
Architecture and associated libraries
2
Visual Tracking
3
Results
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Plan
1
Architecture and associated libraries
2
Visual Tracking
3
Results
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Context
What do we want to do ? A parallel system dedicated to realtime vision applications
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? Bottleneck A classical architecture: bottleneck when image data must be transmit to each node
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The solution A Dual-Bus cluster The solution is to use a dual-bus cluster
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
We propose
A dual-bus Cluster with: a generic architecture model, a dedicated communication bus for video a bus for node communications
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Firefire 400Mbps dedicated for video broadcasting
1 node with 2 CPU
Ethernet switch Gigabit dedicated to nodes communications
Node: XServe PPC G5 Bi-processor SIMD achived by the AltiVec Extention
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Babylon Project
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Associated libraries
Three contributions C+FOX E.V.E. / nT2 Quaff
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Associated libraries
Three contributions C+FOX E.V.E. / nT2 Quaff
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Associated libraries
Three contributions C+FOX E.V.E. / nT2 Quaff
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
C+FOX
Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c img; doSomething(img); c.stop();
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
nT2
nT2
library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.
Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
nT2
nT2
library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.
Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
nT2
nT2
library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.
Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
nT2
nT2
library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.
Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
nT2
nT2
library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.
Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Plan
1
Architecture and associated libraries
2
Visual Tracking
3
Results
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
What do we want to do ?
Visual tracking We want to recognize and track a moving object from a moving camera or a moving stereo basis, at realtime (30fps). We want to track and recognize an object using only a generic model.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
What do we want to do ?
Visual tracking We want to recognize and track a moving object from a moving camera or a moving stereo basis, at realtime (30fps). We want to track and recognize an object using only a generic model.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
A solution ?
using classifiers into a probabilistic tracking framework Bring together recent classifiers (Adaboost, SVM) and a particle filter
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Using Classifiers for Real-Time Tracking: why ?
Offline learning of the object to be tracked Trackers can be designed to track categories of objects (pedestrians, vehicles), Objects are modelized with a collection of views, representing variation of the object appearance, Real Time issue Recent pattern recognition methods works at a near realtime rate (Cascaded Boosting,...)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Using Classifiers for Real-Time Tracking: why ?
Offline learning of the object to be tracked Trackers can be designed to track categories of objects (pedestrians, vehicles), Objects are modelized with a collection of views, representing variation of the object appearance, Real Time issue Recent pattern recognition methods works at a near realtime rate (Cascaded Boosting,...)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Overview of the method t=0 Initialisation where is the object in the first frame ? t++ Prediction Predict the new object position
CLASSIFIER
Observation Update the object position using a likelihood function based on classifier
Particle Filter
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Overview of the method Convert to probabilities
t=0 Initialisation where is the object in the first frame ? t++ Prediction Predict the new object position
? CLASSIFIER
?
Observation Update the object position using a likelihood function based on classifier
Particle Filter
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Observation Function output of a classifier m(f ) Assumption : m(f ) ∈] − ∞; +∞[ with m(f 1 ) < m(f 2 ) −→ P (class|f 1 ) < P (class|f 2 ) Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier : P (positive|m(f )) =
1 1 + exp(A.m(f ) + B)
(1)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
The Observation Function output of a classifier m(f ) Assumption : m(f ) ∈] − ∞; +∞[ with m(f 1 ) < m(f 2 ) −→ P (class|f 1 ) < P (class|f 2 ) Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier : P (positive|m(f )) =
1 1 + exp(A.m(f ) + B)
(1)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Platt scaling Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi ∈ {0; 1}) non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) {− yi log(pi ) + (1 − yi ) log(1 − pi )},
(2)
i
with pi =
1 1 + exp(A.mi + B)
(3)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Platt scaling Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi ∈ {0; 1}) non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) {− yi log(pi ) + (1 − yi ) log(1 − pi )},
(2)
i
with pi =
1 1 + exp(A.mi + B)
(3)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Results 0.7 0.6
0.25
p(x|y = ±1) Neg.
p(x|y = ±1)
Pos.
0.2
Neg. Pos.
0.5 0.15
0.4 0.3
0.1
0.2 0.05 0.1
-10
0
10
20
SVM score (margin)
30
0 0.1
1
0.9
0.9
0.8
0.8
0.7
0.7
0.5 0.4
p(y=1|x)
1
0.6
ements
-20
0.6 0.5 0.4
0.3
0.3
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Adaboost score
0.9
0.1
0.1 0 -30
0.2
p(y=1|x)
0 -30
-20
-10
0
10
20
30
SVM score (margin)
0 0
0.2
0.4
0.6
Adaboost score
0.8
1
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Features Haar based wavelets +1 +1 +1
-1
+1
-1
+1
+1
-1
-1
+1
-1
-1 +1
A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example:For 3 scales and a 128 × 64 pixels image, the number of features is 42560.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Features Haar based wavelets +1 +1 +1
-1
+1
-1
+1
+1
-1
-1
+1
-1
-1 +1
A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example:For 3 scales and a 128 × 64 pixels image, the number of features is 42560.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
features reduction
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
features reduction
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
features reduction
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
features reduction
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Some of the selected features
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Result for a recognition step (Adaboost) 1 0.9 feature vector dimension 50 75 100
0.8 0.7 0.6 0.5
-1
+1
...
...
0.4 0.3 0.2 0.1 00
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Roc curve for three different sizes of feature vector Using a pedestrian database with 2000 samples for training and 2000 samples for testing l
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Result for a recognition step (Adaboost)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Classifier score evolution near the true position of the object
SVM 10
0.8
5 Margin
Score
Adaboost 1
0.6 0.4 0.2 -10 -5 0 5 horizontal trans. (pixels)
0 -5
10
-10 -10 -5 0 5 horizontal trans. (pixels)
10
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )
(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)
(5)
Likelihood function P (Zt |Xt = Xnt ) =
1 n
∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t
o
(6)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )
(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)
(5)
Likelihood function P (Zt |Xt = Xnt ) =
1 n
∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t
o
(6)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )
(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)
(5)
Likelihood function P (Zt |Xt = Xnt ) =
1 n
∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t
o
(6)
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Plan
1
Architecture and associated libraries
2
Visual Tracking
3
Results
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Initialization 1
2
3
The classifier is called for a set of regular positions and scales whitin the image A set of particles is initialized from the positions ans scales associated to the highest outputs of the classifier The number of particles is restricted when the particle distribution is around the object to be tracked. 1/10 particles displayed 600 particles 100 particles
im #001 some particles
im #010
im #020 The number of gravity center of particles is restricted the set of particles
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Comparing SVM and Adaboost classifiers
140 SVM Adaboost
vertical position
horizontal position
120
100
80
60 0
50
100 150 #image (a)
200
250
SVM Adaboost
120 100 80 60 0
100 #image (b)
200
(c) Im#001
(d) Im#261
Video
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Videos (Adaboost)
Particle distribution vehicle tracking Strong variations of the model target switching Learning a specific object
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
N2 Time consuming
Prediction
Observation
Resampling
One iteration
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
N1
Prediction One iteration
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
N1
N2 N1
N14 Prediction
Observation
One iteration
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
N1
wi
N2 N1
N1
N14 Prediction
Observation
Resampling
One iteration
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
N1
wi
N2 N1
N1
N14 Prediction
Observation
Resampling
One iteration
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
3D Tracking
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Parallel implementation
A 3D state vector . Xt = (Pt , Vt )t , where Pt is the 3D position of the center of a bounding box
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
3D Tracking Performances Seq. Par. Gain
200 0.0609s 16.40 0.0231s 43.31 ×2.7
500 0.1439s 6.95 0.0265s 37.72 ×5.42
1000 0.2874s 3.48 0.0313s 31.9 ×9.16
5000 1.6393s 0.61 0.0858s 11.66 ×19.43
10000 3.8462s 0.26 0.1567s 6.38 ×24.5
20 FPS with 2000 particles . Linear evolution of performances according to the number of nodes.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Conclusion We have presented: Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting. Application A Realtime Recognition and tracking method based on the use of detector and combined with a particle filter into a probabilistic framework.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Conclusion We have presented: Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.
and with a dedicated communication bus for video broadcasting. Application A Realtime Recognition and tracking method based on the use of detector and combined with a particle filter into a probabilistic framework.
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi
Architecture and associated libraries
Visual Tracking
Results
Thank you for your attention
For questions in relation with the architecture part, please contact :
[email protected]
Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi