Real-Time Parallel Implementation of a Particle Filter based Visual Tracking Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-Thierry Laprest´ e Lasmea, Clermont-Ferrand, France Lasmea, UMR 6602 CNRS/Blaise Pascal University Clermont-Ferrand, France

May 12, 2006

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Contents

1

Architecture and associated libraries

2

Visual Tracking

3

Results

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Plan

1

Architecture and associated libraries

2

Visual Tracking

3

Results

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Context

What do we want to do ? A parallel system dedicated to realtime vision applications

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? Bottleneck A classical architecture: bottleneck when image data must be transmit to each node

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The solution A Dual-Bus cluster The solution is to use a dual-bus cluster

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

We propose

A dual-bus Cluster with: a generic architecture model, a dedicated communication bus for video a bus for node communications

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Firefire 400Mbps dedicated for video broadcasting

1 node with 2 CPU

Ethernet switch Gigabit dedicated to nodes communications

Node: XServe PPC G5 Bi-processor SIMD achived by the AltiVec Extention

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Babylon Project

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Associated libraries

Three contributions C+FOX E.V.E. / nT2 Quaff

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Associated libraries

Three contributions C+FOX E.V.E. / nT2 Quaff

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Associated libraries

Three contributions C+FOX E.V.E. / nT2 Quaff

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

C+FOX

Video acquisition: C+FOX library (available at sourceforge) Example Camera c(Mode 640x480 Mono8,FPS 30,0); unsigned char* img; c.start(); c.keepAlive(); c  img; doSomething(img); c.stop();

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

nT2

nT2

library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.

Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

nT2

nT2

library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.

Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

nT2

nT2

library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.

Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

nT2

nT2

library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.

Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

nT2

nT2

library (sourceforge as soon as possible) numerical tables manipulation. Matlab like interface which include linear algebra. SIMD intrinsec support.

Implementation Language : C++ Template metaprogrammation. [VELDHUIZEN 96] [ALEXANDRESCU 02]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

QUAFF The QUAFF library Language : C++. efficient method for skeleton design. Simple specification of prototypes.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Plan

1

Architecture and associated libraries

2

Visual Tracking

3

Results

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

What do we want to do ?

Visual tracking We want to recognize and track a moving object from a moving camera or a moving stereo basis, at realtime (30fps). We want to track and recognize an object using only a generic model.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

What do we want to do ?

Visual tracking We want to recognize and track a moving object from a moving camera or a moving stereo basis, at realtime (30fps). We want to track and recognize an object using only a generic model.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Why is it a challenge ? A moving object Variation of the appearence of the object (3D object) Variation of the illumination conditions A moving camera Background/Foreground segmentation can not be done easily to isolate the object. Recognize an object from a generic model This is a object recognition problem Track an object at a realtime framerate This is a temporal tracking problem

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

A solution ?

using classifiers into a probabilistic tracking framework Bring together recent classifiers (Adaboost, SVM) and a particle filter

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Using Classifiers for Real-Time Tracking: why ?

Offline learning of the object to be tracked Trackers can be designed to track categories of objects (pedestrians, vehicles), Objects are modelized with a collection of views, representing variation of the object appearance, Real Time issue Recent pattern recognition methods works at a near realtime rate (Cascaded Boosting,...)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Using Classifiers for Real-Time Tracking: why ?

Offline learning of the object to be tracked Trackers can be designed to track categories of objects (pedestrians, vehicles), Objects are modelized with a collection of views, representing variation of the object appearance, Real Time issue Recent pattern recognition methods works at a near realtime rate (Cascaded Boosting,...)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Using Classifiers for Real-Time Tracking: who SVM S. Avidan, SVT : Support Vector Tracking, IEEE CVPR01. (Using Support Vector Machines (SVM) in an optical flow based tracker. RVM O. Williams A Sparse Probabilistic Learning Algorithm for Real-Time Tracking, ICCV03. (Using Relevance Vector Machine (RVM) in a visual tracking approach, associated to a Kalman Filter) Video Adaboost K. Okuma et al.A boosted particle filter: Multitarget detection and tracking. ECCV04. (Using a weak classifier + Adaboost into the sampling step of a particle filter)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Overview of the method t=0 Initialisation where is the object in the first frame ? t++ Prediction Predict the new object position

CLASSIFIER

Observation Update the object position using a likelihood function based on classifier

Particle Filter

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Overview of the method Convert to probabilities

t=0 Initialisation where is the object in the first frame ? t++ Prediction Predict the new object position

? CLASSIFIER

?

Observation Update the object position using a likelihood function based on classifier

Particle Filter

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Observation Function output of a classifier m(f ) Assumption : m(f ) ∈] − ∞; +∞[ with m(f 1 ) < m(f 2 ) −→ P (class|f 1 ) < P (class|f 2 ) Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier : P (positive|m(f )) =

1 1 + exp(A.m(f ) + B)

(1)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

The Observation Function output of a classifier m(f ) Assumption : m(f ) ∈] − ∞; +∞[ with m(f 1 ) < m(f 2 ) −→ P (class|f 1 ) < P (class|f 2 ) Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier : P (positive|m(f )) =

1 1 + exp(A.m(f ) + B)

(1)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Platt scaling Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi ∈ {0; 1}) non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) {− yi log(pi ) + (1 − yi ) log(1 − pi )},

(2)

i

with pi =

1 1 + exp(A.mi + B)

(3)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Platt scaling Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi ∈ {0; 1}) non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) {− yi log(pi ) + (1 − yi ) log(1 − pi )},

(2)

i

with pi =

1 1 + exp(A.mi + B)

(3)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Results 0.7 0.6

0.25

p(x|y = ±1) Neg.

p(x|y = ±1)

Pos.

0.2

Neg. Pos.

0.5 0.15

0.4 0.3

0.1

0.2 0.05 0.1

-10

0

10

20

SVM score (margin)

30

0 0.1

1

0.9

0.9

0.8

0.8

0.7

0.7

0.5 0.4

p(y=1|x)

1

0.6

ements

-20

0.6 0.5 0.4

0.3

0.3

0.2

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Adaboost score

0.9

0.1

0.1 0 -30

0.2

p(y=1|x)

0 -30

-20

-10

0

10

20

30

SVM score (margin)

0 0

0.2

0.4

0.6

Adaboost score

0.8

1

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Features Haar based wavelets +1 +1 +1

-1

+1

-1

+1

+1

-1

-1

+1

-1

-1 +1

A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example:For 3 scales and a 128 × 64 pixels image, the number of features is 42560.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Features Haar based wavelets +1 +1 +1

-1

+1

-1

+1

+1

-1

-1

+1

-1

-1 +1

A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example:For 3 scales and a 128 × 64 pixels image, the number of features is 42560.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

features reduction

Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

features reduction

Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

features reduction

Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

features reduction

Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Some of the selected features

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Result for a recognition step (Adaboost) 1 0.9 feature vector dimension 50 75 100

0.8 0.7 0.6 0.5

-1

+1

...

...

0.4 0.3 0.2 0.1 00

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Roc curve for three different sizes of feature vector Using a pedestrian database with 2000 samples for training and 2000 samples for testing l

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Result for a recognition step (Adaboost)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Classifier score evolution near the true position of the object

SVM 10

0.8

5 Margin

Score

Adaboost 1

0.6 0.4 0.2 -10 -5 0 5 horizontal trans. (pixels)

0 -5

10

-10 -10 -5 0 5 horizontal trans. (pixels)

10

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )

(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)

(5)

Likelihood function P (Zt |Xt = Xnt ) =

1 n

∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t

o

(6)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )

(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)

(5)

Likelihood function P (Zt |Xt = Xnt ) =

1 n

∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t

o

(6)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

. Particle filter : p(Xt |Z0:t ) = {(Xnt , πtn )}N n=1 State vector . Xt = (ct , ct−1 , st , st−1 )

(4) . with c = (x, y) location of a bounding box whitin the image and associated scale factor st Dynamics Xt+1 = AXt + Bvt , vt ∼ N (0, Σ)

(5)

Likelihood function P (Zt |Xt = Xnt ) =

1 n

∗ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t

o

(6)

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Plan

1

Architecture and associated libraries

2

Visual Tracking

3

Results

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Initialization 1

2

3

The classifier is called for a set of regular positions and scales whitin the image A set of particles is initialized from the positions ans scales associated to the highest outputs of the classifier The number of particles is restricted when the particle distribution is around the object to be tracked. 1/10 particles displayed 600 particles 100 particles

im #001 some particles

im #010

im #020 The number of gravity center of particles is restricted the set of particles

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Comparing SVM and Adaboost classifiers

140 SVM Adaboost

vertical position

horizontal position

120

100

80

60 0

50

100 150 #image (a)

200

250

SVM Adaboost

120 100 80 60 0

100 #image (b)

200

(c) Im#001

(d) Im#261

Video

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Videos (Adaboost)

Particle distribution vehicle tracking Strong variations of the model target switching Learning a specific object

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

N2 Time consuming

Prediction

Observation

Resampling

One iteration

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

N1

Prediction One iteration

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

N1

N2 N1

N14 Prediction

Observation

One iteration

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

N1

wi

N2 N1

N1

N14 Prediction

Observation

Resampling

One iteration

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

N1

wi

N2 N1

N1

N14 Prediction

Observation

Resampling

One iteration

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

3D Tracking

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Parallel implementation

A 3D state vector . Xt = (Pt , Vt )t , where Pt is the 3D position of the center of a bounding box

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

3D Tracking Performances Seq. Par. Gain

200 0.0609s 16.40 0.0231s 43.31 ×2.7

500 0.1439s 6.95 0.0265s 37.72 ×5.42

1000 0.2874s 3.48 0.0313s 31.9 ×9.16

5000 1.6393s 0.61 0.0858s 11.66 ×19.43

10000 3.8462s 0.26 0.1567s 6.38 ×24.5

20 FPS with 2000 particles . Linear evolution of performances according to the number of nodes.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Conclusion We have presented: Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting. Application A Realtime Recognition and tracking method based on the use of detector and combined with a particle filter into a probabilistic framework.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Conclusion We have presented: Architecture A realtime vision parallel architecture with a hybrid three-level parallel programming model: SIMD with altivec, bi-processor distribution of the threads, a high level programming framework for parallel decomposition.

and with a dedicated communication bus for video broadcasting. Application A Realtime Recognition and tracking method based on the use of detector and combined with a particle filter into a probabilistic framework.

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi

Architecture and associated libraries

Visual Tracking

Results

Thank you for your attention

For questions in relation with the architecture part, please contact : [email protected]

Joel Falcou Thierry Chateau, Jocelyn Serrot and Jean-ThierryReal-Time Laprest´ e Parallel Lasmea, Clermont-Ferrand, Implementation of a France Particle Filter based Vi