Formal Representation and Reasoning for Microscopic Medical Image-Based Prognosis. Application to Breast Cancer Grading. Représentation et Raisonnement Formels pour le Pronostic basé sur l’Imagerie Médicale Microscopique. Application à la Graduation du Cancer du Sein. PhD Thesis
Author: Adina Eunice Tutac
Scientific advisors: Prof. Vladimir Ioan Cretu, UPT Prof. Daniel Racoceanu, CNRS Prof. Noureddine Zerhouni, UFC
Timisoara, 22 October 2010
Outline 1. Context • What is Knowledge representation? • Types of Knowledge Representation • Knowledge Representation in Medical Applications. Breast cancer Grading (BCG)
2. My research directions • Define a Methodology for Knowledge Representation in BCG • Model a Breast Cancer Grading Ontology (BCGO) • Integrate BCGO into Cognitive Microscope Framework
3. Conclusions and Future Work 2 of 56
• Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• Knowledge Representation [Davis et al., 1993] –cognition & perception Ø
1. A substitute for the thing itself by which an entity thinks instead of acting- determine consequences
Ø
2. A set of ontological commitment
Ø
3. A fragmentary theory of intelligent reasoning : conception, set of inferences
Ø
4. A medium for pragmatically efficient computation
Ø
5. A language in which we say things about the world 3 of 56
• Context
Ø
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Qualitative representation versus quantitative representation “the aquarium metaphor” [Freksa, 1991]
4 of 56
• Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• Content-Based Image Retrieval (CBIR) Ø
Ø
image similarity-based retrieval [Long et al., 2003] (4-5) image-based reasoning [Sciacio et al., 2002] (3)
O F F L I N E
O N L I N E 5 of 56
• Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• Case-Based Reasoning (CBR) Ø
similar past problems solutions to solve new problems [Aamodt and Plaza, 2004]
(3-5) Ø
experience-based approach (medicine)
6 of 56
• • •
• Context
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Methodology or Technology ? [Tutac, 2009a] CBR
Methodology
Principles
Retrieve
Reuse
Revise
Retain
Technology
Methods & Techniques
Indexing
Retrieval
Refinement
CBIR
Current stage of CBIR
CBIR
Beginning of CBIR
necessary evolution
Learning
7 of 56
• Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
8 of 56
•
Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• What is Description Logics (DL) ? [Baader07] Ø a family of first order logics knowledge representation formalism (e.g. ALC, SHOIN(D), SHIQ (D) ) Ø offers capability of decidability & automated reasoning (3,4)
9 of 56
•
Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• Ontology Web Language (OWL) [McGuiness09] Ø a language to structure the knowledge from real domain of the world (1,2) Ø based on the SH logic family and RDF(S ) Ø variants : OWL Lite, OWL DL, OWL Full
• OWL-DL (SHOIN (D) ) Ø computational completeness Ø high expressivity & decidability power 10 of 56
•
• • •
Context
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Types of ontologies upper-level ontologies –theories of time and space Ø
Ø general
ontologies – intermediate level, task-independent (reference) ontologies – a particular type of the world (medicine) Ø application
ontologies – a specific task
App Ont licatio n olo gy
Ø domain
11 of 56
• Context
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
• Semantic Web Rule Language (SWRL) Ø OWL-DL and RuleML in the same framework [Karimi, 2008]
Ø trade-off : expressivity and decidability
Atom ← C (a) | D(v) | R(a, b)U (a, v) | builtIn( p, v1,..., vn) | a = b | a ≠ b
12 of 56
•
Context
Medical CBIR [Müller et al., 2004] [Quellec et al., 2010]
Medical CBR [Nillson & Sollenborn, 2004] [Schmidt and Gierl, 2001] [Holt et al., 2006]
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Advantages
Drawbacks
Ø increasing rate of image production Ø diagnosis, teaching & research
Ø gaps (semantic, context) Ø relevance feedback Ø page zero problem Ø user interfaces
Ø cognitive adequateness Ø explicit experience Ø duality of knowledge Ø diagnosis, teaching & research
Ø adaptation Ø unreliability Ø concentration on reference
13 of 56
• Context
KR approaches
• • •
Advantages
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Drawbacks
Breast pathology
non-logicbased formalism [Steichen, 2006]
human mind task-solving resemblance
lack of logical inference
semantic networks • large vocabularies (UMLS, SNOMED-CT)
logic-based formalism [Baader et al., 2007]
high expressivity computational power
undecidability in complex representation
Description Logics (DL) • reference ontologies (NCI, GALEN)
14 of 56
• Context
Spatial approaches
• • •
What is Knowledge Representation? Types of Knowledge Representation Knowledge Representation in Medical Applications. BCG
Biomedical ontologies
Advantages
Drawbacks
mereology [Donnelli et al., 2005 ], [Mechouche et al., 2009]
Ø FMA Ø reduces Ø SNOMED-CT ambiguities Ø GALEN Ø symbolic &numerical Ø image processing link
Ø decidability (large vocabularies)
topology [Hudelot et al., 2006] geometry [Mezaris et al., 2004]
Ø FMA (brain MRI images) Ø general purpose
Ø decidability (large vocabularies)
Ø image interpretation Ø reasoning
15 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
SCOR E
• Breast Cancer Grading Ø prognosis assessment tool in modern pathology practice [Steichen et al., 2006]
NUCLEAR PLEOMORPHI SM
Small Regular Uniform Cells Moderate Nuclear Size And Variation Marked Nuclear Variation
1 2
TUBULE FORMATION
Majority of Tumor (>75%) Moderate Degree (10-75%) Little or None ( Mitoses/10 hpf
1 2 3
COMBINED HISTOLOGIC GRADE
Low Grade (I) Intermediate Grade (II) High Grade (III)
3-5 6-7 8-9
Ø a semiological approach Ø Nottingham Grading System
3
16 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
• Breast Cancer Grading problems Ø
time consuming (200 case patients in NUH, Singapore/900 cases in PSH, Paris daily)
Ø
automated image analysis algorithms treat only individual criterion § nuclear pleomorphism score [Adawi et al., 2006] § tubule formation score [Petushi et al., 2006] § mitosis count [Beliën et al., 1997]
Ø
lack of formal knowledge representation (agreement inconsistencies)
Ø
semantic gap 17 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
• concept of similarity • visual content processing & analysis (CBIR) • semantics, ontologies (CBIR) • case-based structure of data (CBR) • duality of medical knowledge (CBR) • incremental learning (CBR) • hybrid reasoning : image-based reasoning & case based reasoning 18 of 56
• My Research Directions
Ø
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
nature of formal representation : qualitative or quantitative? • quantitative approaches force the use of quantities to express qualitative facts [Brageul and Guesgen, 2007] • OWL formalism supports logical qualitative definitions not quantitative definitions • in BCG : “close to neoplasm periphery”, “dividing cell nuclei”, “score 3”
Quantitative definitions are confined to the qualitative representation as numerical values allowed by semantic languages. 19 of 56
• My Research Directions
Ø
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
target of representation • endurants (objects), not perdurants (event, processes) • time-independent spatial representation • application ontology BCGO (1-5) • spatial theory support to eliminate ambiguities and inconsistencies
20 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Ø Two approaches • image driven : semi-automated breast cancer grading
• limitations: no benefit of DL, to close to image level
21 of 56
Explicit knowledge modeling
Mitosis – very dark diving cell nuclei, which are not located in tubule formation area and are close to neoplasm periphery
Formal representation for Mitosis in OWL Class(Mitosis complete NucleusDivision restriction(hasIntensity someValuesFrom VeryLow) restriction(hasIntensity allValuesFrom VeryLow) restriction(isCloseTo someValuesFrom NeoplasmPeriphery ) restriction(isCloseTo allValuesFrom NeoplasmPeriphery ) complementOf(restriction(isLocatedIn someValuesFrom Tubule)) complementOf(restriction(isLocatedIn allValuesFrom Tubule)) restriction(hasEccentricity hasValue hasEccentricityValue))
Pathologist Knowledge acquisition NCI thesaurus
Knowledge translation
Structural Modeling (OWL-DL) Rule modeling (SWRL)
Pellet reasoner
Knowledge refining
Expert feedback
• semantic-driven [Uschold and Grüninger, 1996]
c Formal representation for Mitosis in OWL - DL Mitosis ≡ NucleusDivisionΠ ∃hasIntensity.VeryLowΠ ∀hasIntensity.VeryLowΠ ∃isCloseTo.NeoplasmPeriphery Π ∀isCloseTo.NeoplasmPeriphery Π ¬∃isLocatedIn.TubuleΠ ¬∀isLocatedIn.TubuleΠ ∍ hasEccentricity has hasEcentricityValue Formal representation for hasEccentricityValue in SWRL Nucleus (? y ) ∧ hasEccentricity (? x,? value) ∧ swrlb : lessThan(? value, 1) ∧ swrlb : greaterThan(? value, 0 ) → hasEccentricityValue(? x,? value) 22 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
1. Knowledge acquisition Ø ontology semi-automated segmentation (PROMPT) Ø a knowledge base KA =(NCI, NGS) NCI → National Cancer Institute thesaurus (e.g. Disease, Patient, Assessment, Specimen) NGS → Nottingham Grading System & 20 patients cases NUH (e.g. NottinghamGrading, Tubule, Mitosis) Ø Why NCI and not SNOMED-CT or UMLS? • • •
NCI is homogenous, dedicated to cancer information representation semantic mapping and concept alignment NCI is freely available 23 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
2. Knowledge translation [Tutac, 2009b] Ø structural modeling with OWL-DL (high expressivity)
Ø rule modeling with SWRL - DL safe rules (decidability) 24 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
3. Knowledge refining Ø Pellet reasoner • • •
based on the DL tableau algorithm verifies ontology consistency computes inferred knowledge
Ø Medical feedback
25 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Spatial Reasoning. Formal theory user query Spatial Concepts Mereo-topologic Metric/Geometric Dimensional
Spatial claims
Spatial Reasoning Manual Automated
Spatial inference results
Spatial Representation Axioms Theorems 26 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Spatial Reasoning. Mereo-topological axioms & theorems Definition. Surrounded By (SurrBy) [Tutac et al., 2010]
( SA1) SurrBy ( x, y ) B Pr( x)r ( y ) ∧ ~ PCoin( x, y )) or ( SA2) SurrBy ( x, y ) B Loc - In( x, y ) ∧ ~ O( x, y ) Applied to classes:
( ST 1) SurrBy1 ( A, B) B ∀x( Inst ( x, A) → ∃y ( Inst ( y, B) ∧ SurrBy( x, y)) ( ST 2) SurrBy2 ( A, B) B ∀y ( Inst ( y, B) → ∃x( Inst ( x, A) ∧ SurrBy ( x, y )) ( ST 3) SurrBy12 ( A, B) B Loc - In1 ( A, B) ∧ Loc - In2 ( A, B) 27 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Spatial Reasoning. Mereo-topological axioms & theorems How to eliminate ambiguities or redundant definitions ? Definition. Inclusion (Included-In)
( IA1) Included − In( x, y) B ∃zLoc − In( x, z) ∧ Loc − In( y, z)∧ ~ PCoin( x, y)) If PCoin(x,y) denotes overlapping, and SurrBy is already defined as previously
Included-In is redundant 28 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Spatial reasoning, Metric axioms/theorems RCC-8 & composition table DC(x,y) x
EC(x,y) y
CloseTo(x,y)
TPP(x,z) x
x
z
z
y
CloseTo(x,y) y
TPP(x,z) x
x
z
z
EC(y,z)
EC(y,z) y
y
x
z
y
z
(CA1)CloseTo( x, y ) B DC ( x, y ) ∨ EC ( x, y ) ∧ TPP ( x, z ) ∧ EC ( y, z ), where z80.000 frames ) Ø resolution issue (mitosis are identified on a different scale than tubules) => precision/recall on different scales/ Mean-Average Precision? Ø the DCIS issue
47 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Model applicability. MICO framework [Roux et al., 2009a] MICO – Project funded recently by ANR, TecSan 2010
48 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Ontology-driven mitosis and tubule formation scoring
49 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
Ontology-based retrieval and validation support
50 of 56
• My Research Directions
• • •
Methodology for Knowledge Representation in BCG Modeling BCG Ontology BCGO integration into Cognitive Microscope Framework
MICO prototype based on CBIR-CBR paradigm
51 of 56
• Conclusions & Future Work
• •
Contributions Perspectives
Ø CBIR –CBR methodological comparative analysis Ø approach to bring together image and semantic representation (perception and cognition) :breast cancer grading application ontology, qualitative representation, spatial integration, modeling of perdurants § bridges
semantic and context gap : formal logic semantic indexing techniques § offers high expressivity and decidability : OWL-DL and SWRL formalisms § semantic reasoning : DL reasoning extended with spatial reasoning § semantic retrieval for evaluation and validation of the ontology
Ø integration of ontology in a cognitive virtual microscope framework following the CBIR-CBR methodology 52 of 56
• Conclusions & Future Work
• •
Contributions Perspectives
Ø refining of ontology (e.g. scale information, temporal extension) Ø explanation-based medicine Ø integration in reference ontologies (NCI thesaurus vs SNOMED-CT, UMLS, MeSH) Ø evaluation in a clinical setting (intra-observer and inter-observer agreement, k-coefficient) § define methodology § define set of metrics § apply metrics Ø Integration in complex representation with heterogeneous datainteroperability (VPH) 53 of 56
Selected Publications
•
[Tutac et al., 2009a] A.E. Tutac, D. Racoceanu, T.Putti, W-K.Leow, H.Muller, T.Putti, V.Cretu, “Towards Translational Incremental Similarity Based in breast cancer grading”, in Medical Imaging 2009: Computer-Aided Diagnosis, Proc. SPIE (SPIE, Bellingham, WA 2009), Vol. 7260, 72603C (2009), pp.1-12, Nico Karssemeijer, Maryellen L.Giger eds, ISBN : 978-0-8194-7511-4, Orlando, Florida, SUA, 7-12 February 2009, in “Progress in biomedical optics and imaging, ISSN: 1602-1744, SUA (IEEE, CAT-INIST indexed)
•
[Tutac et al., 2008] A.E. Tutac, D. Racoceanu, T. Putti, W. Xiong, W.K. Leow, and V. Cretu, “Knowledge-Guided Semantic Indexing of Breast Cancer Histopathology Images”, BioMedical Engineering and Informatics: New Development and the Future, in Proc. BMEI, ed.Yonghong Peng and Yufeng Zhang pp. 107-112, China, 2008 (ISI, IEEE indexed)
•
[Roux et al., 2009a] L. Roux, A. Tutac, N. Lomenie, D. Balensi, A. Veillard, D. Racoceanu, W.K. Leow, J. Klossa, T.C. Putti, “A cognitive virtual microscopic framework for knowledgebased exploration of large microscopic image in breast cancer histopathology”, in Proc. EMBC, vol.1, pp.3697-702, 2-6 Sept, Minneapolis, SUA (IEEE indexed)
•
[Roux et al., 2009b] Roux L., Tutac A., Veillard A., Dalle J., Racoceanu D., Lomenie N., Klossa J, "A cognitive approach to microscopy analysis applied to automatic breast cancer grading" , Virchows Archiv The European Journal of Pathology, Springer-Verlag Berlin Heidelberg, H.Höfler ed, no. 428, vol. 455, suppl 1, pp.34-35, ISSN : 0945-6317 (Print) 1432-2307 (Online), 22nd European Congress of Pathology, Florence, Italy, Sept 2009 (ISI indexed) 54 of 56
Selected Publications
•
[Tutac et al., 2009b] A. Tutac, D.Racoceanu, N.Loménie , W.K.Leow., L.Roux, V.I.Cretu, TPutti , "Knowledge Modeling of Breast Cancer Grading using OWL-DL formalism", Virchows Archiv The European Journal of Pathology, Springer-Verlag Berlin Heidelberg, H. Höfler ed, no. 428 vol. 455, suppl 1, pp. 36, ISSN : 0945-6317 (Print) 1432-2307 (Online), 22nd European Congress of Pathology, Florence, Italy, 4-9 Sept 2009 (SpringerLink indexed)
•
[Tutac et al., 2009c] Adina Tutac, Daniel Racoceanu, Nicolas Loménie, Ludovic Roux, Thomas C. Putti, Vladimir Cretu, “Breast Cancer Grading Knowledge Modeling and Reasoning for Cognitive Virtual Microscopy”, National Institutes of Health NIH Inter-Institute Workshop on Optical Diagnostic and Biophotonic Methods from Bench to Bedside, Bethesda, USA, 1- 2 Oct 2009
•
[Tutac et al., 2009d] Adina Tutac, Daniel Racoceanu, Nicolas Loménie, Ludovic Roux, Didier Balensi and Thomas Putti, “Knowledge Representation and Reasoning for Breast Cancer Grading in Cognitive Virtual Microscope Framework”, A*STAR Scientific Conference 2009, Biopolis, Singapore, 28-29 Oct, 2009
•
[Lomenie et a., 2009] N.Loménie, L.Roux, D. Balensi, A. Tutac, D. Racoceanu, “MICO: The COgnitive Virtual Microscope project”, Cognitive Systems with Interactive Sensors (COGIS) symposium, Paris, France, 16-18 Nov 2009
•
[Tutac et al., 2010] A. Tutac, V.Cretu, D.Racoceanu “Spatial representation for Breast Cancer Grading Ontology", Proc. IEEE International Joint Conferences on Computational Cybernetics and Technical Informatics ICCC-CONTI, pp.89-94, Timisoara, Romania, 27-29 May, 2010 (IEEE indexed) 55 of 56
Research Activity
•
Research grants/internships : Ø
TD- 65/2008 “Micro-Medical Image Processing”
Ø
“HISTOGRAD – a virtual microscope for breast cancer grading” Patent - software declaration (inventoried as *DI 2944-01* by the CNRS for the *UMI 2955*. Registered by the CNRS, Daniel Racoceanu, Adina Tutac, Xiong Wei, Jean-Romain Dalle, Chao-Hui Huang, Ludovic Roux, Wee-Kheng Leow)
Ø
CNRS, France & NUS, Singapore - 3 research stages in Singapore 2007 - 2009
56 of 56
