Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Tree Graph Views for a Distributed Pervasive Environment Tuyˆet Trˆam DANG NGOC (
[email protected]) Nicolas TRAVERS (
[email protected]) ETIS Laboratory, University of Cergy-Pontoise
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Outline
1
Context
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment
4
PADAWAN Prototype
5
Conclusion Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
1
Context Motivation Data Integration Issues
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment
4
PADAWAN Prototype
5
Conclusion
PADAWAN Prototype
Conclusion
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Motivation
Context Integrate the pervasive vision of computing devices in order to allow any user anywhere in the world to query anything from anywhere in the world. Ad−Hoc Network Fireman
Sensor Network
Expert
Doctor Officer
Proxy
PADAWAN Fireman
Sink
Emergency Unit
802.11 AP
IP Router
Internet
0 1 1 0 0 1
LDAP Primergy
News Feeder Proxy
SGBD
Web Server
PADAWAN Département des Sciences Informatiques
GIS
Client
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Motivation
Context Integrate the pervasive vision of computing devices in order to allow any user anywhere in the world to query anything from anywhere in the world. Ad−Hoc Network Fireman
Sensor Network
Expert
Doctor Officer
Proxy
PADAWAN Fireman
Sink
Emergency Unit
802.11 AP
IP Router
Internet
0 1 1 0 0 1
LDAP Primergy
News Feeder Proxy
SGBD
Web Server
PADAWAN Département des Sciences Informatiques
GIS
Client
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Motivation
Context How to organize the graph, so a user with any type of profile can connect on any node of the graph and access to other data that are dissiminated on nodes of the graph. Nodes Heterogeneous autonomous datasources Proxies (both client and source) Heterogeneous client applications Heterogeneous Network Links
C
P
S
C
S
S S
S
C
C
C
S S
C
P
Legend
S
P
C S
S Data Source
C
Client Terminal
P
Proxy Wire Link Wireless Link
S
S
S
C
P C S
P
C
S
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues
Source Heterogeneous Data (relational, semi-structured) →common query language and model : XML/XQuery
SELECT p.name, p.address FROM person p, car c WHERE age>60 AND p.id=c.driver AND c.color LIKE "RED"
GET http://example.com/persons.xml GET http://example.com/car.xml
ORDER BY p.address
name
Doeuf
Well
Database
address
Cergy
Well Rose 80 Paris Cover ...
Paris
Web Site
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues XQuery
Source Heterogeneous Data (relational, semi-structured) →common query language and model : XML/XQuery + Wrapper
XML
Wrapper
Database
XQuery
XML
Wrapper
Web Site
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues
SELECT p.name, p.address FROM person p, car c WHERE age>60 AND p.id=c.driver AND c.color LIKE "RED"
Source Autonomous Sources queries more or less supported
GET http://example.com/persons.xml GET http://example.com/car.xml
ORDER BY p.address
Well Rose Here are the whole documents, 80 make the processing yourself ! Paris name
Doeuf
address
Cergy
Cover ...
Here are the exact results Well
Database
Paris
Web Site
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues CPU(Join)=card(T1) *card(T2)*CPU(restr)*TIO CPU (restr)=0.0001ms TIO=0.000282ms
Cost model ?
Cost model ? Tps(statistic)=3ms (Avg)
Tps(Calibration)=4*CPU(Join)
Source Autonomous Sources queries more or less supported , internal information not available (cost model, statistics, schema) →Source Description
Wrapper
Cost model ?
CPU(Join)=card(T1) *card(T2)*CPU(restr)*TIO CPU (restr)=0.0001ms TIO=0.000282ms
Wrapper
Cost model ?
?
Database Database
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues
Client Heterogeneity access permission, user preferences, terminal capabilities →Client profile + View
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues Client
Data integration evaluate query on distributed node →mediation + Source description
Mediator
Wrapper
Wrapper
Wrapper
Wrapper
Source
Source
Source
Source
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Data Integration Issues
Data Integration Issues Source Heterogeneous Data (relational, semi-structured) →common query language and model : XML/XQuery + Wrapper Autonomous Sources queries more or less supported, internal information not available (cost model, statistics, schema) →Source Description
Client Heterogeneity access permission, user preferences, terminal capabilities →Client profile + View Data integration evaluate query on distributed node →mediation + Source description
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
1
Context
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment
4
PADAWAN Prototype
5
Conclusion
PADAWAN Prototype
Conclusion
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
TGV : Tree Graph View
PADAWAN Prototype
Conclusion
[ Travers and Dang-Ngoc 2004-2007 ]
declare function local:f($doc as xs:string) as element() {for $x in (doc(?rev.xml?)/review|doc(?$doc?)/catalog) [. contains(?Robin Hobb?)]/book/[.//price > 15] where some $y in $x/comments satisfies contains ($y, ?Excellent?) order by $x/@isbn return {$x/@isbn} {$x//price/text()} { if (count($x/title) > 2) then { for $z in doc(?books.xml?)/book where $z/@isbn = $x/@isbn return {($z/title)[3]} }else } }
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
TGV : Tree Graph View
PADAWAN Prototype
Conclusion
[ Travers and Dang-Ngoc 2004-2007 ]
XQuery has a (too) rich syntax : XPath Constraints, Filter Quantifiers Document construction Nesting, Aggregates, Sorts Conditional operators, Set operators Sequences, Function Hard to parse, process and identify subpart and dependancies. XQuery
XML
Query Processor Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
TGV : Tree Graph View
PADAWAN Prototype
Conclusion
[ Travers and Dang-Ngoc 2004-2007 ]
Model based on Tree Pattern matching that : Supports Full untyped XQuery [ WEBIST2007 ] Provides an intuitive representation [ DASFAA 2007 ] Designed for distributed environment [ IBIS Journal 2006 ] Support for optimisation and for evaluation [ ICEIS 2007 ] XQuery
XML
Rule Processing
Canonized XQuery logical TGV
Application profile
Annotated TGV
Optimization
TGV Evaluation annotation
Evaluation Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Example ”list every buildings occupied by more than 100 inhabitants, and for each, get the district and the list of maximum temperature measured by the sensors located in the same district.” for $a in /buildings/building where $a/description/inhabitant > 100 return {$a/district} { for $b in //sensor where $b/deploymentArea/district = $a/district return {$b/max temp} }
Yellow Lake 14 Green Valley 163 25 43 Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Example ”list every buildings occupied by more than 100 inhabitants, and for each, get the district and the list of maximum temperature measured by the sensors located in the same district.” for $a in /buildings/building where $a/description/inhabitant > 100 return {$a/district} { for $b in //sensor where $b/deploymentArea/district = $a/district return {$b/max temp} }
buildings $a
$b
building
description
district =
sensor
deploymentArea
inhabitant > 100
max_temp
district
$t temperature location districtMonitoring temperatures
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Generic Annotation for any type of Information A layer per information type Information on any set of TGV elements (including annotation) Any granularity, possible recovering Generic Annotation (ex. several type of cost model with complex formulas)
1) Physical TGV with location annotation
2) Physical TGV with cost annotation
buildings $a building description district Source1 inhabitant > 100
sensor $b Source2, Source 3 =
Mediator
buildings location Mediator $a building deploymentArea temperatures description district = inhabitant > 100 CS1=Cost(S1)
deploymentArea max location
$t sensor $b sensor deploymentArea max location
C1=CS1*sel*CS2*IO
Cost(S2)
$t
location buildings cp=max(C1, deploymentArea $a temperatures building cost=cp+Op description district inhabitant Yellow 3) Physical TGV > Lake 100 | 1678 Rhode Forest | 1986 with evaluation Green Valleybuildings | 82761 annotation $a location building deploymentArea temperatures description district inhabitant > 100
4) Logical TGV location deploymentArea temperatures
C2)
C2=cost_card*0.02 sensor
$b sensor deploymentArea max location Yellow Lake | 14 Green Valley| 163 $t Green Valley|sensor 25 $b sensor Green Valley| 43 deploymentArea max = location =
$t sensor
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules
Rule Annotated TGV Pattern condition”
→
Annotated TGV Pattern Conclusion”
Rules for :
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules card ($x)*card($y)*CPU_unit CPU_unit=0,02
=
=
Rules for : Local Transformation
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules person
50,000 card
name
age
refsrc
person name
age
refsrc FR_SRC2
Rules for : Local Transformation
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules card ($x) > card ($y)
card ($y)
bind−join
$x
=
card ($x)
bind−join $y
$y
=
$x
Rules for : Local Transformation Optimization
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules join (eval ($a), eval ($b)) $a
=
$b
$a
=
$b
Rules for : Local Transformation Optimization Local Evaluation Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV : Rules
Rules for : Local Transformation Optimization Local Evaluation Distributed Evaluation Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
1
Context
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment TGV Evaluation TGV*
4
PADAWAN Prototype
5
Conclusion
PADAWAN Prototype
Conclusion
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV Mobile Agent Initiated by the client Evaluated on the fly Parallel evaluation :
S
cloning the TGV on some nodes. evaluation rules located on the node.
Leaves of the recursion tree on source node. Source node fill the evaluation annotation layer.
S
TGV
S TGV
TGV
S S P
TGV
S P TGV
S S C
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV Mobile Agent Initiated by the client Evaluated on the fly Parallel evaluation :
S TGV eval TGV eval
cloning the TGV on some nodes. evaluation rules located on the node.
Leaves of the recursion tree on source node. Source node fill the evaluation annotation layer.
S
S TGV eval
S S P
TGV eval
S P TGV eval
TGV eval
S S C
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV : Evaluation Yellow Lake Green Valley Green Valley Green Valley
Yellow Lake | 678 Rhode Forrest| 1986 Green Valley | 82761
Evaluation rules : Evaluation on Source Wrappers
| 14 | 163 | 25 | 43
buildings $a
$b
building description
district =
sensor
deploymentArea
inhabitant > 100
max_temp
district
$t temperature location districtMonitoring temperatures
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV : Evaluation
Evaluation rules : Evaluation on Source Wrappers Join
Yellow Lake Green Valley Green Valley Green Valley
| 678 | 14 | 82761 | 163 | 82761 | 25 | 82761 | 43
buildings $a
$b
building description
district =
sensor
deploymentArea
inhabitant > 100
max_temp
district
$t temperature location districtMonitoring temperatures
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV : Evaluation
Evaluation rules : Evaluation on Source Wrappers Join Nest
Yellow Lake Green Valley Green Valley Green Valley
| 678 | 14 | 82761 | 163 | 82761 | 25 | 82761 | 43
buildings $a
$b
building description
district =
sensor
deploymentArea
inhabitant > 100
district
$t temperature location districtMonitoring
max_temp
14 163 25 43
temperatures
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV Evaluation
TGV : Evaluation
Evaluation rules : Evaluation on Source Wrappers Join Nest Final projection
Yellow Lake | 678 Green Valley | 82761 | |
| 14 | 163 | 25 | 43
buildings $a
$b
building description
district =
sensor
deploymentArea
inhabitant > 100
max_temp
district
$t temperature location districtMonitoring temperatures
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
TGV*
TGV* : Sources Description
How to annotate the TGV ? →TGV* to describe sources in an exhaustive manner. Dataguide : source part (STP) of the TGV with annotation Annotations Set of rules (for operators)
Location
person car name adress id app street town imm model brand col personn car id app streettownimmmodelbrandcol
Cost (memory) nameadress
Rules Join (cost−memoryà
person
car
Cost (exec_time) nameadress
id app streettownimmmodelbrandcol
personn
Dataguide
Restriction (cost exec_ti Join (cost−memory)
car
nameadress id app streettownimmmodelbrandcol
TGV* Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
1
Context
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment
4
PADAWAN Prototype Nodes and Agents
5
Conclusion
PADAWAN Prototype
Conclusion
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Nodes and Agents
PADAWAN : a Multi-Agent Plateform Client TGV Forwarders
Oracle Wrapper
2
1
TGV *
Profile 3
Node 1 Forwarders Rules Cache TGV *
Wrappers
Rules
4
Node 4 Forwarders
Node 2
Web Site Wrapper TGV *Client Forwarders TGV
Node 3
ProfileProfile 6
Node 0
Node 7 Rules
5
Sensor Wrapper Forwarders TGV *
Node 6
It is the agents located on the node that define the functionalities of the node.
Node 5
Client Node : Client and Presentation Agents Source Node : Wrapper Agent Proxy Node : Cache, Router, and Rules Agents Repository Node : Agent Library Prototype PADAWAN (a Plateform for All Devices Accessing the World And Neighbourhood) on http ://padawan.ensea.fr/
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
1
Context
2
TGV : Tree Graph View
3
TGV for a Pervasive Environment
4
PADAWAN Prototype
5
Conclusion Bibliography Simulation
PADAWAN Prototype
Conclusion
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Conclusion TGV Mobile Agent Full untyped-XQuery evaluation Evaluation through the Multi-Agents Platform PADAWAN graph Dynamic Optimization Current Work : TGV* Mobile Agent describe data generic enough to describe all information known on a source. abstract TGV* Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Bibliography
Publications on TGV and PADAWAN Dang-Ngoc, T.-T. and Travers, N. (2007). Tree graph views for a distributed pervasive environment. In the 1st International Conference on Network-Based Information Systems (NBIS), Regensburg, Germany. Liu, T., Dang-Ngoc, T.-T., and Laurent, D. (2007). Cost framework for a distributed semi-structured environment. In in the proceedings of the International workshop Database Management and Application over Networks DBMAN (APWeb/WAIM Workshop), Huangshan, China. Travers, N. and Dang-Ngoc, T.-T. (2007). An extensible rule transformation model for xquery optimization. In The 9th International Conference on Enterprise Information Systems (ICEIS), Madeira, Portugal. Travers, N., Dang-Ngoc, T.-T., and Liu, T. (2006). Tgv : an efficient model for xquery evaluation within an interoperable system. International Journal of Interoperability in Business Information Systems (IBIS), 3. ISSN : 1862-6378. Travers, N., Dang-Ngoc, T.-T., and Liu, T. (2007a). An efficient evaluation of xquery with tgv. In the 3rd International Conference of WEB Information Systems and Technologies (Web-IST), Spain. Travers, N., Dang-Ngoc, T.-T., and Liu, T. (2007b). Full untyped xquery canonisation. In in the proceedings of the International workshop on Emerging Trends of Web Technologies and Applications -WebETrends (APWeb/WAIM Workshop), Huangshan, China. Travers, N., Dang-Ngoc, T.-T., and Liu, T. (2007c). Tgv : a tree graph view for modelling untyped xquery. In the 12th International Conference on Database Systems for Advanced Applications (DASFAA), Thailand.
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Discrete Event Simulation
4a− r(C1,P1,P2) 4b− r(C1,P1,P2)
P
C
S1 3a−R(C1,P1,P2)
C
P P2
3b− R(C1,P1,P2)
2a− R(C1,P1)
S2
1− R(C1)
C1
P1
2b− R(C1,P1)
6− r(C1)
S3
SINK
C
3c−
r(C1,P1)
5a−
r(C1,P1)
S
S
Sources already located located route of the request randomly generated, using randomly 0-N sources time of service, and queries generation parameterized Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation (a) Mediation Infrastructure nbclients = 3
nbproxy = 1
S1
C1
Mediation Architecture Direct Access to sources Bottleneck on mediator
nbsources = 6
S2 C2
P1
S3 S4
C3
S5 S6
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation (b) Infrastructure PàP nbclients = 3
P2P Network Each Peer is a mediator Direct Access to sources Load on each peer Each peer knows how to resolve a whole request
nbproxy = 5
nbsources = 6 S1
C1
P1
S2
P2
S3
P3
S4
P4
S5
P5
S6
C2
C3
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation Padawan Network A peer has access to some other peers and eventually to some sources Distribution of the query evaluation Evaluation depending on the peers capabilities Distributed waiting time
(c) Infrastructure PADAWAN nbclients = 3
nbproxy = 5
nbsources = 6 S1
C1
P3 P1
S2
P5 C2 S3
P2
P4 S4
C3
S5 S6
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation Result
Times / Number of queries to process in the system 20 queries generator (=clients), 100 sources, For P2P and PADAWAN : 20 peers. Mediator : Bottleneck P2P Architecture better thant PADAWAN Architecture Linear
graph
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation Result Times / Number of queries to process in the system 20 queries generator (=clients), 100 sources, For P2P and PADAWAN : 20 peers. Mediator : Bottleneck P2P Architecture better thant PADAWAN Architecture Linear But : Mean waiting time better in PADAWAN
graph-tw
Département des Sciences Informatiques
Context
TGV : Tree Graph View
TGV for a Pervasive Environment
PADAWAN Prototype
Conclusion
Simulation
Simulation limitation Opening connection delay PADAWAN : few connection per proxy →all connections can be left opened. P2P : each peer can potentially reach any source →can’t maintain all opening/closing connection →consider connection delay
Nodes capabilities In the major P2P networks, equivalent nodes with same capabilities (or 2 or 3 categories with Superpeer and normal peer) PADAWAN queries can be evaluated on nodes with different capabilities Département des Sciences Informatiques
