Incremental construction of a proximity graph for large image collections exploration Frédéric Rayar, Sabine Barrat, Fatma Bouali and Gilles Venturini Université François Rabelais de Tours Computer Science Laboratory
Big Data Mining and Visualisation June 19, 2015 - Lyon
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
2
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation Context PhD goals
2
Incremental construction of a proximity graph
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
3
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Context Big Data • Search • Summarise • Visualise
Incremental construction of a proximity graph for large image collections exploration
4
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Context Big Data • Search • Summarise • Visualise
Large image collection • Open Access Images (e.g. museums, art galleries) • Social networks (e.g. Facebook, Instagram) • Medical images
Incremental construction of a proximity graph for large image collections exploration
5
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Retrieval vs. exploration Image retrieval • Concept-based: keywords, annotations • Content-based: visual descriptors (CBIR)
⇒ Locality-sensitive hashing (LSH)
Incremental construction of a proximity graph for large image collections exploration
6
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Retrieval vs. exploration Image retrieval • Concept-based: keywords, annotations • Content-based: visual descriptors (CBIR)
⇒ Locality-sensitive hashing (LSH) Visualisation • Global distribution • Local neighbourhood
Incremental construction of a proximity graph for large image collections exploration
7
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Retrieval vs. exploration Image retrieval • Concept-based: keywords, annotations • Content-based: visual descriptors (CBIR)
⇒ Locality-sensitive hashing (LSH) Visualisation • Global distribution • Local neighbourhood
Exploration/navigation of a collection: ⇒ having insights, extracting knowledge
Incremental construction of a proximity graph for large image collections exploration
8
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
PhD goals Indexing Index the image collection in a relevant structure: • Incremental • Content and/or concept-based description
Incremental construction of a proximity graph for large image collections exploration
9
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
PhD goals Indexing Index the image collection in a relevant structure: • Incremental • Content and/or concept-based description
Visualisation Visualise the image collection: • Interactive • User feedback
Incremental construction of a proximity graph for large image collections exploration
10
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
PhD goals Indexing Index the image collection in a relevant structure: • Incremental • Content and/or concept-based description
Visualisation Visualise the image collection: • Interactive • User feedback
Incremental construction of a proximity graph for large image collections exploration
11
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph Relative neighbourhood graph (RNG) Existing work First proposed approach Second proposed approach Experiments
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
12
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Image graph Image graph • Nodes = Images • Edges if two nodes are similar
Incremental construction of a proximity graph for large image collections exploration
13
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Image graph Image graph • Nodes = Images • Edges if two nodes are similar
Proximity graph • Introduced by G. Toussaint [Tou91] • Weighted graph with no loop • Extract the structure of a data point set D ⊂ Rd • Edge between two points of D if they are close enough
Incremental construction of a proximity graph for large image collections exploration
14
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Proximity graph (PG) Notable PG • k-nearest neighbour graph (k-NNG) • relative neighbourhood graph [Tou80] (RNG) • Gabriel graph [GS69] (GG) • Delaunay graph [Del34] (DG)
Incremental construction of a proximity graph for large image collections exploration
15
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Proximity graph (PG) Notable PG • k-nearest neighbour graph (k-NNG) • relative neighbourhood graph [Tou80] (RNG) • Gabriel graph [GS69] (GG) • Delaunay graph [Del34] (DG)
⇒ 1-NNG(D) ⊂ RNG(D) ⊂ GG(D) ⊂ DG(D) [Urq82]
Incremental construction of a proximity graph for large image collections exploration
16
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Proximity graph (PG) Notable PG • k-nearest neighbour graph (k-NNG) • relative neighbourhood graph [Tou80] (RNG) • Gabriel graph [GS69] (GG) • Delaunay graph [Del34] (DG)
⇒ 1-NNG(D) ⊂ RNG(D) ⊂ GG(D) ⊂ DG(D) [Urq82]
Incremental construction of a proximity graph for large image collections exploration
17
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Proximity graph (PG) Notable PG • k-nearest neighbour graph (k-NNG) • relative neighbourhood graph [Tou80] (RNG) • Gabriel graph [GS69] (GG) • Delaunay graph [Del34] (DG)
⇒ 1-NNG(D) ⊂ RNG(D) ⊂ GG(D) ⊂ DG(D) [Urq82] Focus on RNG • Sparse graph • Connected graph
Incremental construction of a proximity graph for large image collections exploration
18
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
RNG Definition By definition, p 6= q ∈ D are relative neighbours if and only: ∀r ∈ D\{p, q}, δ(p, q) ≤ max(δ(p, r), δ(q, r)) where δ : D × D → R is a distance function. r p
q
Incremental construction of a proximity graph for large image collections exploration
19
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
RNG Algorithm Input: D Output: RNG = (V , E) 1: V = D; E = ∅ 2: for each p ∈ V do 3: for each q ∈ V do 4: for each r ∈ V do 5: if δ(p, q) ≤ max(δ(p, r), δ(q, r)) then 6: E = E ∪ {pq} 7: end if 8: end for 9: end for 10: end for 11: return RNG = (V , E) Incremental construction of a proximity graph for large image collections exploration
20
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph Relative neighbourhood graph (RNG) Existing work First proposed approach Second proposed approach Experiments
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
21
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
Incremental construction of a proximity graph for large image collections exploration
22
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
q
Incremental construction of a proximity graph for large image collections exploration
23
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
q nn
Incremental construction of a proximity graph for large image collections exploration
24
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
q fn
nn
Incremental construction of a proximity graph for large image collections exploration
25
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
q nn
fn
q fn
nn
Incremental construction of a proximity graph for large image collections exploration
26
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
sr
q fn
nn
Incremental construction of a proximity graph for large image collections exploration
27
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
SR=0 q fn
nn
Incremental construction of a proximity graph for large image collections exploration
28
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
SR=0 q fn
nn
Incremental construction of a proximity graph for large image collections exploration
29
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
SR=1
q fn
nn
Incremental construction of a proximity graph for large image collections exploration
30
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Illustration
SR=1
q fn
nn
Incremental construction of a proximity graph for large image collections exploration
31
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Main steps Incremental RNG construction Let us consider RNG = (V , E) of n points and a new point q to insert: 1
nn = nearest neighbour of q in V
2
fn = farthest relative neighbour of nn in V
3
sr = (δ(q, nn) + δ(nn, fn)) ∗ (1 + )
4
SR = p ∈ V , δ(p, q) ≤ sr
5
Update SR with the O(n 3 ) algorithm
Insertion complexity = O(2n + n 03 ), with n 0 = |SR|.
Incremental construction of a proximity graph for large image collections exploration
32
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Limitations Influence of • Set empirically at = 0.1 [HY07]
Incremental construction of a proximity graph for large image collections exploration
33
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Limitations Influence of • Set empirically at = 0.1 [HY07] • Missing relative neighbours of q
fn nn
q
g SR
Incremental construction of a proximity graph for large image collections exploration
34
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Limitations Influence of • Set empirically at = 0.1 [HY07] • Missing relative neighbours of q • False relative neighbours not invalidated in SR
fn nn
SR
q
g SR
g
Incremental construction of a proximity graph for large image collections exploration
35
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Hacid et al. 2007 - Limitations Influence of n 0 • Insertion complexity O(2n + n 03 ), with n 0 = |SR| • n 0 0).
q N1 (q) N1e (q) N2 (q) N2e (q) Incremental construction of a proximity graph for large image collections exploration
40
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Algorithm Edge-based neighbourhood local update strategy Let us consider RNG = (V , E) of n points and a new point q to insert: 1
nn = nearest neighbour of q in V
2
fn = farthest relative neighbour of nn in V
3
sr = (δ(q, nn) + δ(nn, fn)) ∗ (1 + )
4
SR = p ∈ V , δ(p, q) ≤ sr
Incremental construction of a proximity graph for large image collections exploration
41
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Algorithm Edge-based neighbourhood local update strategy Let us consider RNG = (V , E) of n points and a new point q to insert:
5
Compute relative neighbours of q in SR [O(n 02 )]
6
Compute edge-based neighbourhood of q [O(deg L )]
7
(In)validate edges in the neighbourhood w.r.t. q
Incremental construction of a proximity graph for large image collections exploration
42
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Algorithm Edge-based neighbourhood local update strategy Let us consider RNG = (V , E) of n points and a new point q to insert:
Insertion complexity = O(2n + n 02 + deg L ), with n 0 = |SR|, deg the average degree and L the order of the neighbourhood. Incremental construction of a proximity graph for large image collections exploration
43
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph Relative neighbourhood graph (RNG) Existing work First proposed approach Second proposed approach Experiments
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
44
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Datasets Datasets Available on the online UCI machine learning repository [BL13]. D Iris WDBC Breiman
|V| 150 569 5000
d 4 30 40
| E(RNG) | 195 712 17,837
Incremental construction of a proximity graph for large image collections exploration
45
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Datasets Datasets Available on the online UCI machine learning repository [BL13]. D Iris WDBC Breiman Corel68k MF-1M [HL08]
|V| 150 569 5000 68,040 1,000,000
d 4 30 40 57 150
| E(RNG) | 195 712 17,837 190,410 n.t.
Incremental construction of a proximity graph for large image collections exploration
46
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Accuracy Accuracy Number of wrongly added edges and removed edges in the RNGs computed.
| E(RNG) | Iris WDBC Breiman Corel68k
195 712 17837 190410
Algorithm 2 +10/-2 +2/-1 +0/-0 +20363/-11
L=2 +8/-1 +10/-0 +1161/-0 +9089/-356
Algorithm 4 L=3 == +3/-0 +299/-0 +2165/-388
Incremental construction of a proximity graph for large image collections exploration
L=4 == == +26/-0 +637/-397
47
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Accuracy
% wrongly added or removed edges
Accuracy Percentage of wrongly added or removed edges over L. Iris WDBC Breiman Corel68k
6
4
2
1 0 2
3 neighbourhood order
Incremental construction of a proximity graph for large image collections exploration
4
48
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Time computation Time computation Comparison of the computation times of Algorithms 2 and 4 (in seconds). Algorithm 2 Breiman Corel68k MF-1M
7692 122h >> 250h
Algorithm L=2 L=3 16 25 889 1371 145h 151h
Incremental construction of a proximity graph for large image collections exploration
4 L=4 178 1604 181h
49
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Time computation Time computation Insertion times distribution over L for Corel68k and MIRFLICKR-1M (in seconds).
Incremental construction of a proximity graph for large image collections exploration
50
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Publication (under review) Rayar, F., Barrat, S., Bouali, F., and Venturini, G. (2015). An approximate proximity graph incremental construction for large image collections indexing.
Incremental construction of a proximity graph for large image collections exploration
51
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph
3
Visualisation Existing solutions Prototype
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
52
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Workflow
www
Collecte
image
Description
Structuration
Visualisation
(EHD)
(RNG)
(Tulip, Gephi)
ehd
graphe
meta donnée
Incremental construction of a proximity graph for large image collections exploration
53
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Dataset NGA Images • Open Access Images • 43721 images from National Gallery of Art (Washington) • Edge Histogram Descriptor (EHD) : dimension 80 • 117 925 edges created in the RNG
Incremental construction of a proximity graph for large image collections exploration
54
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Dataset NGA Images • Open Access Images • 43721 images from National Gallery of Art (Washington) • Edge Histogram Descriptor (EHD) : dimension 80 • 117 925 edges created in the RNG
Graph drawing algorithms Software: Tulip & Gephi • OpenOrd (2011) [Gephi] • Yifan Yu multi-niveau (2005) [Gephi] • FM3 (2005) [Tulip]
Incremental construction of a proximity graph for large image collections exploration
55
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Software
Incremental construction of a proximity graph for large image collections exploration
56
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Gephi - OpenOrd
Incremental construction of a proximity graph for large image collections exploration
57
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Gephi - Yifan Yu multilevel
Incremental construction of a proximity graph for large image collections exploration
58
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Tulip - FM3
Incremental construction of a proximity graph for large image collections exploration
59
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Tulip - FM3 (zoomed)
Incremental construction of a proximity graph for large image collections exploration
60
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Observations [Rayar et al., 2015] Questions • Descriptor choice • Graph drawing algorithm choice • Images as node • Slow interaction
Publication Rayar, F., Barrat, S., Bouali, F., and Venturini, G. (2015). Exploration visuelle et interactive d’une large collection d’images en libre accès. In EGC 2015 - Atelier VIF. Incremental construction of a proximity graph for large image collections exploration
61
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph
3
Visualisation Existing solutions Prototype
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
62
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Web-based image graph exploration platform
Main issue • Memory management (image loading) • Easy local access for image
Incremental construction of a proximity graph for large image collections exploration
63
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Demonstration afterwards if you are interested!
Incremental construction of a proximity graph for large image collections exploration
64
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Sommaire 1
Motivation
2
Incremental construction of a proximity graph
3
Visualisation
4
Perspectives
Incremental construction of a proximity graph for large image collections exploration
65
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Perspectives Indexing • Multilevel approach • Content and/or concept-based description
Visualisation • Platform improvement • Multilevel interface • User evaluation • User feedback
Incremental construction of a proximity graph for large image collections exploration
66
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
Thanks for your attention! Questions?
Incremental construction of a proximity graph for large image collections exploration
67
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
References I K. Bache and M. Lichman, UCI machine learning repository, http://archive.ics.uci.edu/ml, 2013. B. N. Delaunay, Sur la sphère vide, Bulletin of Academy of Sciences of the USSR 7 (1934), 793–800. R. K. Gabriel and R. R. Sokal, A New Statistical Approach to Geographic Variation Analysis, Systematic Zoology 18 (1969), no. 3, 259–278. Mark J. Huiskes and Michael S. Lew, The mir flickr retrieval evaluation, MIR ’08: Proceedings of the 2008 ACM International Conference on Multimedia Information Retrieval (New York, NY, USA), ACM, 2008.
Incremental construction of a proximity graph for large image collections exploration
68
Motivation
Incremental construction of a proximity graph
Visualisation
Perspectives
References II Hakim Hacid and Tetsuya Yoshida, Incremental neighborhood graphs construction for multidimensional databases indexing, Canadian Conference on AI, 2007, pp. 405–416. G. T. Toussaint, The relative neighbourhood graph of a finite planar set, Pattern Recognition 12 (1980), 261–268. , Some unsolved problems on proximity graphs. R. Urquhart, Graph theoretical clustering based on limited neighbourhood sets, Pattern Recognition 15 (1982), no. 3, 173 – 187.
Incremental construction of a proximity graph for large image collections exploration
69
