I could not come to you but ...

International Meteor Conference at Armagh Observatory All welcome! 16-19 September, 120 / 155 euro Organizers: David Asher, Tolis Christou, Mark Bailey, Geert Barentsen

http://www.imo.net/imc2010

Northern Ireland Non-erupting extinct volcanos

Virtual Meteor Observatory ●

●

After the initial reduction of observations, how to exploit and archive the data? Outline ●

Architecture

●

Data model

●

Ingesting data

●

Querying the VMO

●

Recent developments

Architecture Metrec2XML UFOAnalyzer

Add / query data XML/ZIP

Scriptable Interface (HTTP)

PostgreSQL Database

Other Software

PHP (Drupal) Browser Interface Browse data Registration of codes Administration User

Usage

Camera stations

Science Radio stations VMO

Fireball reports

Realtime orbits

Visual reports

MPC

Dreaming ...

Realtime flux

Alert

Known NEO? Pointing strategy

Multiple similar orbits?

NEO Precovery

Sky surveys NEO Discovery

●

●

Priority: standardize data formats and protocols. We could aim for a central database, but multiple databases (nodes) are also OK when standards exist. IMO Node User

Application

Standards Other Node?

●

Standards should be agreed by a "VMO Team" (not owned by a particular group/organization) ●

cf. ISSI meetings

●

Currently MetRec+Visual data - so we called it "IMO node"7

●

Outline ●

Architecture

●

Data model

●

Ingesting data

●

Querying the VMO

●

Recent developments

8

Data model ●

Data model = collection of tables linked together ●

●

●

Stored in a "relational SQL database" to allow fast and easy read/write operations and to ensure consistency. XML used for transport. Focus on reduced science data. Raw data is less important to be in a standard form.

Initial approach: generic model for video/visual/radio/fireball ●

●

One "meteor table" and "session table" to fit EVERYTHING Implemented in 2006 as an object-oriented "Unified Meteor Database" during my master studies (computer science)

●

But ... abstract database was complex and confusing ●

Science data should be stored in its natural form. -> VMO uses different tables for different techniques

● ●

We achieved version 1.0 of the data model at ISSI meeting. SonotaCo took the time to write down useful comments. I annotated his slides - we could discuss and produce version 1.1.

Model: Camera Observations cam_session

1 N

Location Observer System Configuration

1

Start / Stop time Effective time Field Clouds Limiting mag.

N

cam_meteor Appearance time Begin / End position Speed Max. brightness Image / Video

cam_period

1 N

cam_pos Time Position Brightness

Generic for video and still (< 1 fps ?) cameras

Stored as tables in database ...

Corresponding XML format

Visualization: data browser - http://vmo.imo.net

Model: Camera Components Not standardized (yet?) - not directly necessary for computations

cam_system cam_lens cam_prism cam_session

cam_intensifier cam_camera cam_digitizer cam_software

Model: Visual Reports 1

vis_session

1

Location Observer N 1

vis_rate_period Start / Stop time Effective time Limiting magnitude Field of view Obstruction

N

vis_magnitude_period Shower Limiting magnitude

vis_magnitude_bin N

vis_rate_shower Shower Number

Magnitude Number

N

1

Much improved Visual Form - Rainer: please demonstrate later

Data model - Entry codes ●

We need "Primary Keys" to identify entries

●

Fixed scheme for data:

●

●

Session = CAM-20100417-LIC1

●

Meteor = CAM-20100417-LIC1-M001

●

Visual report = VIS-20100417-ARLRA

Non-fixed for objects: ●

System = VOLCANOCAM1

●

Camera = MTV-12V6HC-EX

●

Lens = COM0608

(convention: uppercase alphanumeric without spaces)

Data browser uses „Drupal“ framework with custom PHP-based modules - Source code of modules is well documented and uses simple configuration files

●

Outline ●

Architecture

●

Data model

●

Ingesting data

●

Querying the VMO

●

Recent developments

●

Protocol for adding data 1) Send XML/ZIP file to a VMO webservice ("HTTP POST" protocol) 2) Data added to VMO with status "UNCHECKED" 3) Webservice responds about result Below: toy example works but needs more proper specification

Ingesting data: Quality Flag UNCHECKED

Validation

HOLD

Realtime products

OK

Science

Delete

Flags set by team of administrators using automated tools/interfaces (e.g., Rainer for visual, Sirko for MetRec, SonotaCo for UFO)

Example: Ingesting MetRec

Ingestion procedure was implemented for MetRec using „MetRec2XML“

Metrec2XML ●

JAVA-based software: 1. Reads MetRec *.log and *.inf using regular expressions 2. Performs validation checks 3. Produces XML file for VMO

●

●

Tested in 2008 by converting the entire MetRec archive ●

Successful

●

Large number of double-station meteors found when queried

Updated in 2009 after ISSI ●

MetRec needs to be ingested again using new data model

MetRec Data Flow 1. Upload session (directory or ZIP-file) Observer

vmo.imo.net

sftp://vmo.imo.net/METREC/OBSERVER_CODE/SYSTEM_CODE/YYYYMMDD

2. Quality control

User

Validate

DB

Upload correction Detlef, can you demonstrate - now or later?

●

Outline ●

Architecture

●

Data model

●

Ingesting data

●

Querying the VMO

●

Recent developments

Manual query via user interface e.g. find double-station meteors and download necessary data for double-station computation in CSV-format.

Scriptable query, e.g., http://vmo.imo.net/vo/meteors/DATE=2009-08-12 (also CSV and dBase formats available, see Query -> Products)

Scriptable query using VO protocol e.g.: http://vmo.imo.net/vo/meteors?DATE=2009-08-12

Direct query Using Python (similar to IDL, Matlab, Perl ...) # Example VMO query import pg db = pg.connect(host="vmo.imo.net", port=5432, dbname="vmo", user="imo", passwd="immy") result = db.query("SELECT mag FROM cam_meteor WHERE time::date = '2009-08-12'::date AND shower_code = 'PER'") # Plot a magnitude histogram hist(mag)

●

Outline ●

Architecture

●

Data model

●

Ingesting data

●

Querying the VMO

●

Recent developments

Developments last months ●

Updated data model after ISSI

●

Added "operational" features

●

●

Automated nightly unit tests

●

Logging of database changes and errors

●

Notification e-mails

Progress in recent months slow ...

PhD funded until next year - requires 99% of my dedication to succeed ...

[Barentsen et al. 2010]

●

I am dedicated this year to: ●

Update the VMO data format using SonotaCo input (V1.1)

●

Transfer the Visual database

●

●

●

With Rainer, as soon as possible.

●

Status: needs more testing + implement ZHR graphs

Get the MetRec archive operational ●

With Sirko & Detlef

●

Should be ready for use

Currently little spare time is left to define the scriptable interfaces, support new software, work on automated orbits, etc... small team of programmers would be good