International Meteor Conference at Armagh Observatory All welcome! 16-19 September, 120 / 155 euro Organizers: David Asher, Tolis Christou, Mark Bailey, Geert Barentsen
http://www.imo.net/imc2010
Northern Ireland Non-erupting extinct volcanos
Virtual Meteor Observatory ●
●
After the initial reduction of observations, how to exploit and archive the data? Outline ●
Architecture
●
Data model
●
Ingesting data
●
Querying the VMO
●
Recent developments
Architecture Metrec2XML UFOAnalyzer
Add / query data XML/ZIP
Scriptable Interface (HTTP)
PostgreSQL Database
Other Software
PHP (Drupal) Browser Interface Browse data Registration of codes Administration User
Usage
Camera stations
Science Radio stations VMO
Fireball reports
Realtime orbits
Visual reports
MPC
Dreaming ...
Realtime flux
Alert
Known NEO? Pointing strategy
Multiple similar orbits?
NEO Precovery
Sky surveys NEO Discovery
●
●
Priority: standardize data formats and protocols. We could aim for a central database, but multiple databases (nodes) are also OK when standards exist. IMO Node User
Application
Standards Other Node?
●
Standards should be agreed by a "VMO Team" (not owned by a particular group/organization) ●
cf. ISSI meetings
●
Currently MetRec+Visual data - so we called it "IMO node"7
●
Outline ●
Architecture
●
Data model
●
Ingesting data
●
Querying the VMO
●
Recent developments
8
Data model ●
Data model = collection of tables linked together ●
●
●
Stored in a "relational SQL database" to allow fast and easy read/write operations and to ensure consistency. XML used for transport. Focus on reduced science data. Raw data is less important to be in a standard form.
Initial approach: generic model for video/visual/radio/fireball ●
●
One "meteor table" and "session table" to fit EVERYTHING Implemented in 2006 as an object-oriented "Unified Meteor Database" during my master studies (computer science)
●
But ... abstract database was complex and confusing ●
Science data should be stored in its natural form. -> VMO uses different tables for different techniques
● ●
We achieved version 1.0 of the data model at ISSI meeting. SonotaCo took the time to write down useful comments. I annotated his slides - we could discuss and produce version 1.1.
Model: Camera Observations cam_session
1 N
Location Observer System Configuration
1
Start / Stop time Effective time Field Clouds Limiting mag.
N
cam_meteor Appearance time Begin / End position Speed Max. brightness Image / Video
cam_period
1 N
cam_pos Time Position Brightness
Generic for video and still (< 1 fps ?) cameras
Stored as tables in database ...
Corresponding XML format
Visualization: data browser - http://vmo.imo.net
Model: Camera Components Not standardized (yet?) - not directly necessary for computations
vis_rate_period Start / Stop time Effective time Limiting magnitude Field of view Obstruction
N
vis_magnitude_period Shower Limiting magnitude
vis_magnitude_bin N
vis_rate_shower Shower Number
Magnitude Number
N
1
Much improved Visual Form - Rainer: please demonstrate later
Data model - Entry codes ●
We need "Primary Keys" to identify entries
●
Fixed scheme for data:
●
●
Session = CAM-20100417-LIC1
●
Meteor = CAM-20100417-LIC1-M001
●
Visual report = VIS-20100417-ARLRA
Non-fixed for objects: ●
System = VOLCANOCAM1
●
Camera = MTV-12V6HC-EX
●
Lens = COM0608
(convention: uppercase alphanumeric without spaces)
Data browser uses „Drupal“ framework with custom PHP-based modules - Source code of modules is well documented and uses simple configuration files
●
Outline ●
Architecture
●
Data model
●
Ingesting data
●
Querying the VMO
●
Recent developments
●
Protocol for adding data 1) Send XML/ZIP file to a VMO webservice ("HTTP POST" protocol) 2) Data added to VMO with status "UNCHECKED" 3) Webservice responds about result Below: toy example works but needs more proper specification
Ingesting data: Quality Flag UNCHECKED
Validation
HOLD
Realtime products
OK
Science
Delete
Flags set by team of administrators using automated tools/interfaces (e.g., Rainer for visual, Sirko for MetRec, SonotaCo for UFO)
Example: Ingesting MetRec
Ingestion procedure was implemented for MetRec using „MetRec2XML“
Metrec2XML ●
JAVA-based software: 1. Reads MetRec *.log and *.inf using regular expressions 2. Performs validation checks 3. Produces XML file for VMO
●
●
Tested in 2008 by converting the entire MetRec archive ●
Successful
●
Large number of double-station meteors found when queried
Updated in 2009 after ISSI ●
MetRec needs to be ingested again using new data model
MetRec Data Flow 1. Upload session (directory or ZIP-file) Observer
Upload correction Detlef, can you demonstrate - now or later?
●
Outline ●
Architecture
●
Data model
●
Ingesting data
●
Querying the VMO
●
Recent developments
Manual query via user interface e.g. find double-station meteors and download necessary data for double-station computation in CSV-format.
Scriptable query, e.g., http://vmo.imo.net/vo/meteors/DATE=2009-08-12 (also CSV and dBase formats available, see Query -> Products)
Scriptable query using VO protocol e.g.: http://vmo.imo.net/vo/meteors?DATE=2009-08-12
Direct query Using Python (similar to IDL, Matlab, Perl ...) # Example VMO query import pg db = pg.connect(host="vmo.imo.net", port=5432, dbname="vmo", user="imo", passwd="immy") result = db.query("SELECT mag FROM cam_meteor WHERE time::date = '2009-08-12'::date AND shower_code = 'PER'") # Plot a magnitude histogram hist(mag)
●
Outline ●
Architecture
●
Data model
●
Ingesting data
●
Querying the VMO
●
Recent developments
Developments last months ●
Updated data model after ISSI
●
Added "operational" features
●
●
Automated nightly unit tests
●
Logging of database changes and errors
●
Notification e-mails
Progress in recent months slow ...
PhD funded until next year - requires 99% of my dedication to succeed ...
[Barentsen et al. 2010]
●
I am dedicated this year to: ●
Update the VMO data format using SonotaCo input (V1.1)
●
Transfer the Visual database
●
●
●
With Rainer, as soon as possible.
●
Status: needs more testing + implement ZHR graphs
Get the MetRec archive operational ●
With Sirko & Detlef
●
Should be ready for use
Currently little spare time is left to define the scriptable interfaces, support new software, work on automated orbits, etc... small team of programmers would be good
