ESIEA Meteosat Second Generation User Station Handbook Louis CHABARDÉS ESIEA

[email protected] January 17, 2005

Contents Introduction

3

I

4

MSG : Formats, standards, and inescapables actors

1 Actors 1.1 1.2 1.3 1.4 1.5

Eumetsat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MeteoFrance . . . . . . . . . . . . . . . . . . . . . . . . . . . . Research Group (Groupement de recherche in French) Prodig Hardware Supplier (SIS) . . . . . . . . . . . . . . . . . . . . . Former Members . . . . . . . . . . . . . . . . . . . . . . . . .

2 Formats et Channel 2.1 2.2 2.3 2.4

HRIT . . . . . . . . . . . . . . . . . . Incoming pictures from other satellites: Indirect Reception: . . . . . . . . . . . Data Encryption . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

3 Size, rate Calculation: Necessary disk space Estimation 3.1 3.2 3.3 3.4

II

Number of images by hours . . . . . . . . . . Size of an image: . . . . . . . . . . . . . . . . Amount of storage for a day, a month, a year: Bzip2 compression . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . . . . . .

. . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

Hardware View of the Station

5.1 5.2 5.3 5.4

Hardware consideration . Antenna . . . . . . . . . Satellites coordinates . . Compatibility . . . . . .

5 5 6 6 6

7

7 8 8 9

10

10 10 10 11

12

4 Device Overview 5 Reception

5

13 . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

1

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

14

14 14 14 14

CONTENTS

6 Traitement

15

7 Storage

16

8 Network

18

III

19

6.1 Computing power consideration . . . . . . . . . . . . . . . . . . . . . . . . 15 6.2 Hardware needs: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 7.1 Hardware Charasteristics: . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 7.2 Advices for using such a solution . . . . . . . . . . . . . . . . . . . . . . . 17 7.3 RAID level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Software view of the station:

9 Overview

20

10 Network Conguration

22

9.1 Formats Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 9.2 Tasks Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 9.3 Logging & locking System . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Page 2

PAIR MeteoSat Second Generation

January 17, 2005

CONTENTS

Introduction The aim of this document is to present in an accessible way, every concepts that are needed to build a Meteosat Second Generation User Station. In this way, we will present the general principles of the satellite funtionning. But also, in a more practical way: every hardware and software aspects of the High Rate User Station.We hope this document will allow those who didn't follow the entire project, to be able to build such a station. Although this document is intended to non computer specialists, it is recommended to have a minimum background in : Linux installation, IP based networking, and of courses some basis in Satellite Imaging Systems. I specially would like to thank Julien Cornebise and his companions of PAIR MSG 2004, Laurent Beaudoin for his availability, and Peter Wilson for the way he allow me to compose this document. I hope this document will evolve as all the components of the station will receive upgrades. I f you detect errors, or spelling mistakes, don't hesitate to announce it by mail, at the following adress: [email protected]

Page 3

PAIR MeteoSat Second Generation

January 17, 2005

Part I MSG : Formats, standards, and inescapables actors

4

CHAPTER 1.

ACTORS

Chapter 1 Actors We tried to list here, every persons that matter for the PAIR MSG group, from the students, to the hardware suppliers, but also searchers and miscellanous contacts. You can nd here according each of them, their website, e-mail adress, or even phone numbers. If you ever need help for something related to the content of this document, don't hesitate to ask for help.

1.1 Eumetsat Eumetsat is the European Organisation for the Exploitation of Meteorological Satellites. We didn't have any direct contact with this organisation, but their website is a real goldmine when you are looking for technical documentation regarding any of the Meteosat satellites. In this way, we strongly recommend this URL: www.eumetsat.de You can access the technical document archives by the link named "Publication" in the left of the page. There you can access all technical documents in the PDF format. Be careful, using this site with another web-browser than Microsoft Internet Explorer, is quite unpredictable.

1.2 MeteoFrance For every administrative procedures, we didn't have to face directly with Eumetsat, but with the national entity in charge in France: Meteofrance. In Great Britain, you will have to contact Met Oce, each country have his own related organisation. The complete list is available on Eumetsat's Web site. The main duty of Meteofrance for us is to carry about our "MSG Image Data Services User Registration Form". This document has to be lled by any group that want to access to the MSG Image Data Services. In this form, you have to list the options that you want to access (we will see below what it means), the usage you will make of the datas, etc. If you applies their condition, you will be allowed to receive the whole datas of MSG. MSG datas are encrypted, so it is necessary to have an Eumetsat Decryption Key. We will see further the practical aspect of this encryption.

Page 5

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 1.

ACTORS

1.3 Research Group (Groupement de recherche in French) Prodig Since the beginning of this project, we are in contact with academicians in the domain: Jean Claude Bergès from the Paris 7 university, and Franck Chopin from the Dynamic Meteorology Laboratory of Polytechnic (LMD). The aim of this group is to make easier the access to the satellite datas for the participants. It is necessary to know that there is only two HRUS in France: the rst is owned by Prodig, the second is the one we develop at E.S.I.E.A.. The GDR Prodig also creates operationnal methods for using Meteosat Images. • Web site of Prodig: http://prodig.univ-paris1.fr/msg/ • Data archives website: http://msg.univ-paris7.fr/ • J.-C. Bergès (P1-P7): [email protected] • F. Chopin (LMD, CNRS): [email protected]

1.4 Hardware Supplier (SIS) To obtain all the hardware parts that we needed for the Receiving Chain, we use the hardware supplier of our school SIS. This contact was helpful because we had to nd very specic parts. But we will see further that because of the evolution of the computer technologies, any hardware suppliers can apply this job. We will see later what can be expected for hardware evolutions.

1.5 Former Members I can't end this section without mentionning Julien Cornebise, member of PAIR MSG in 2004, because his help and interest have allowed this project to pass through years, and made an easier transition for PAIR MSG 2005. He is the most entousiastic person I have ever seen in this project, so don't hesitate: • E-mail: [email protected] • Phone Number: 06 08 36 29 13

Page 6

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 2.

FORMATS ET CHANNEL

Chapter 2 Formats et Channel Meteosat Second Generation provides a brand new way of seeing the Earth with Meteorological Imaging. He can provides high resolution pictures on 12 spectral channels, allowing in this way to imagine complete new applications. Let's see in detail what it is about. When you use an High Rate User Station (HRUS), you can access to both High Rate Information Transmission (HRIT) and Low Rate Information Transmission (LRIT) data. The dierence is mainly the lower resolution, and only 5 spectral channel for LRIT service. Let's see in detail the characteristics of the HRIT service.

2.1 HRIT As you seen it earlier, MSG provides a set of images in the 12 spectral channels every quarter. The spectral channel are: • HRV: High Resolution Visible • VIS 0.6|0.8: Visible with a dierent wavelenght • IR 1.6|3.9|6.2|7.3|8.7|9.7|10.8|12.0|13.4: Infrared with dierent wavelenght corre-

sponding to Water Vapour, Dioxide, etc.

All channels except HRV have the following resolution: 3712 (E-W) by 3712 (N-S). It implies a sampling distance of 3 kilometers at sub-satellite point. The HRV channel provides the impressive resolution of 11136 (N-S) by 5568 (E-W). The sampling distance is 1 kilometers at sub-satellite point. For every channel, the pixel depth is 10 bit, whereas it is 8 bit for LRIT. The raw datas that you receive from the satellite aren't directly operational: you need to apply on it a process. Each pictures is segmented into 64 segments plus the Data header and the Data Trailer. There is several reason for this organisation: • Segmented datas are easier to check: you can implement checksum procedures in

order to be sure that any data alteration didn't happened during the transfert.

Page 7

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 2.

FORMATS ET CHANNEL

• The Spinning Enhanced Visible and IR Imager (SEVIRI) boarded on the satellite,

use the satellite spin to take one line of the image at each round. Each segment correspond to a denite number of cycle.

In addition to the pure image datas, an HRUS also receives service messages. They are regrouped under the name "Satellite Application Facilities" (SAF). They are seven, and each have a specic goal: • Short Time Forecasting: Cloud Information, Precipitation Statistics, Wind Vectors • Ocean and Sea SAF: Surface Informations like Wind Vectors, Temperature, ice edge,

etc.

• Ozone Monitoring SAF: distribution aerosols and ultraviolets datas. • Climate Monitoring SAF: Cloud Data Surface Radiation Budget, Water Vapour

Layer Information.

• Numerical Weather Prediction SAF: With the help of powerful computing, provide

forecast from a few hours up to ten days.

• Land Surface Analysis SAF: Albedo Aerosols Radiance Field, Temperature, Snow

Cover.

2.2 Incoming pictures from other satellites: A HRUS can also relay data from other satellites. For instance, when a exceptionnal events happened like cyclone, typhoon, etc, images received from satelittes of other organisation are relayed. You can identify them with their special header. These satellites cover the western part of the atlantic and the Americas. These images are obtained from GOES-E, GOES-W, GOMS and GMS.

2.3 Indirect Reception: A High Rate User Station doesn't aim directly MSG. Datas are received by the Primary Ground Station (PGS) in Usingen. Then those datas are transfered for processing in Darmstadt, at Eumetsat Headquarters: MSG Mission Control Center (MCC). The main operations that are processed in these facilities are: • line-by-line examination • re-sampling in order to re-align the data send by each set of detectors, these pertu-

bations comes from the movement of the spacecraft.

• calibration • radiometric and geometric quality test

Page 8

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 2.

FORMATS ET CHANNEL

Then, all processed segments are send to the PGS in order to be re-transmitted to users. The User's station points on the Hotbird satellite (also used for television), which relays all processed datas.

2.4 Data Encryption As we seen it earlier, Image Data are encrypted. To exploit, use, spread satellite pictures, you need Eumetsat agreement. However, every 6 hours, a complete set of data is unencrypted, and leaved to the public domain.

Page 9

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 3.

SIZE, RATE CALCULATION: NECESSARY DISK SPACE ESTIMATION

Chapter 3 Size, rate Calculation: Necessary disk space Estimation It is necessary to know the global amount of storage that we need to keep all images in a year. The main reason is that the storage is the main problem of the chain. Having a good knowledge of all sizes we work with, will allow us to have a better management of our datas.

3.1 Number of images by hours As we already know, we receive one complete set of images every quarter. We also know that we receive 12 spectral channel, because we are in the case of an HRUS, storing only the HRIT datas. Then the result is: 12 × 4 = 48 images an hour

3.2 Size of an image: The nominal resolution of an image is: 3712 × 3712 for every channel except HRV. And the pixel depth is 10 bit. So the size is: 3712 × 3712 × 10 = 137789440 bits or 17223680 Bytes. 17 MBytes for each images. This is only the theorical result. But in fact, images are stored in the PGM format, and the pixel depth for this format is 16 bit. The new result with this depth give: 3712 × 3712 × 16 = 220463104 Bit or 27557888 Bytes. Finally the real size of an images is 27 MBytes.

3.3 Amount of storage for a day, a month, a year: A simple calculus gives us: 27 × 12 × 4 = 1296 MBytes an hour 1296 × 24 = 31104 Mbytes a day, or 31, 104 GBytes 31104 × 365 = 11352960, or 11352, 60 GBytes, or 11, 352 TBytes

Page 10

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 3.

SIZE, RATE CALCULATION: NECESSARY DISK SPACE ESTIMATION

It is a huge amount of storage, unreachable with a reasonnable budget. That why we use lossless compression with Bzip2.

3.4 Bzip2 compression As we seen it in the last section, storing the whole amount of data produced by MSG is crazy, and it would be a waste of money to store the raw data. The easiest solution to counter this problem is to use a lossless compression for images. We can't accept data loss during compression. The gain obtained by Bzip2 compression is about 72%. For instance the le IR_016-200408281200.PGM has an original size of 27553072 Bytes. After applying a bzip2 compression, the le has a new size of 4992242 Bytes. The exact gain is 71.8%. A projection on all amount of datas for a year gives: (18.1 × 11352960)/100 = 2054775 MBytes, or a little more than 2 Tbytes.

Page 11

PAIR MeteoSat Second Generation

January 17, 2005

Part II Hardware View of the Station

12

CHAPTER 4.

DEVICE OVERVIEW

Chapter 4 Device Overview For the chain conception, we mainly focused on availability, and interoperability. That's why we only use standards X86 hardware. Another advantage of using classical parts is the easiness of replacing malfunctionning parts. Each process is assigned on separate machines: Reception, Processing, and Storage are executed on three dierent computer. Let's see in detail each machine:

Page 13

PAIR MeteoSat Second Generation

January 17, 2005

Chapter 5 Reception

5.1 Hardware consideration The rst operation is to receive datas from the satellite antenna. The needs for this step is simple: a standards X86 PC is sucient. Of course, it must have a digital DVB acquisition PCI card. There is no need for a high computing power because the only task of this machine is to receive pictures and move it to the Processing Computer. For instance, we use a DELL 2.8 GHz Desktop computer, with 256 MBytes of RAM. In addition, a minimum of 200 GBytes of hard-disk space is a good idea. It allow buering in case of failure further in the chain. Of course, a 10/100 networking capability is essential.

5.2 Antenna This computer is linked to the satellite antenna, so keep in mind to place it near, or envisage to have a long cable starting from the antenna. I f you already have an antenna in your facility, it is possible to install a double head in order to recover your signal, without stopping the reception of the existing one. This solution avoid bandwidth issues that implies any other solution.

5.3 Satellites coordinates The coordinates of the Hotbird satellite is 13 degres East. This is a largely used television satellite. That's why we talked about the dual head.

5.4 Compatibility Actually, the reception process runs under Windows, so any Windows compatible computer is compliant. But recently, Eumetsat deliver an Linux based solution for this step. So if you want a 100% Linux based solution, be sure that your hardware is Linux compliant.

14

Chapter 6 Traitement

6.1 Computing power consideration Receveid images segments must be processed to be usable. We will see it later, but this step needs a lot of computing power. The whole treatment must be done in less than 15 minutes, if not, treatment is not, the new set of data is received, and then the whole chain is delayed. Our estimation is that a Pentium 4 clocked at 2.8 GHz, carry out the calculus in 7 minutes. We also tried with a Celeron 1.8 GHz, that was pretty low, and it is better to keep an advance in order to be sure.

6.2 Hardware needs: This step runs totally under Linux. So any Linux compliant hardware is sucient: • AMD Athlon or Intel Pentium 4 >2.5 GHz • At least 512 Mbytes of RAM • 100 GBytes of hard-disk storage • 10/100 Ethernet Networking Capability

15

Chapter 7 Storage In order to store about 2 TeraBytes, dierent solutions are possible: • Using a SAN solution:

A SAN is a all-in-one box that supply network accessible storage. THis type of solution is used in cutting-edge solution for its speed, its robustness. However, the price of this solutions is too high for a small entity as PAIR MSG.

• Several simple computer using distributed storage:

Some free software developpers have written a software solution that assembles the le system of each machines to provide a unique net le system. These system are named CodaFS, Intermezzo. The problem of these solution is that they are completely experimental. We can't bare this type of uncertainties.

• Completely customised case:

12 250 GigaBytes hard-disk, plugged on a hardware-RAID controller card. That's what we used.

This choice has been dictated by many reason: it oers a good balance between price, exibility and security. But hardware evolution is so fast, that now, it is possible to buy professional SAN of 3 TBytes at an aordable price.

7.1 Hardware Charasteristics: We used a CFS 2000S cases to hold every hardware parts

http://www.amselectronics.com/index-CF2000S.html Inside, we choose to use: • A Rioworks HDAMA motherboard http://www.rioworks.com/HDAMA.htm • AMD 244 Opteron processor • 2 512 Corsair DDR-Ram ECC Registered Memory Modules • One 8506-12 ports Serial ATA Raid Controller http://www.3ware.com/products/ serial_ata8000.asp

16

CHAPTER 7.

STORAGE

• One 74 GBytes "Raptor" Maxtor Serial ATA Hard-Disk • 12 Maxtor 250 GBytes Serial ATA hard-disk • One Enermax 660 W EPS 12 V Power Units

This is a beautiful machine, but be careful, integration of such a machine is quite dicult. Be sure that your hardware supplier is worthy of condence. For instance, we had problems with the motherboard, and without a hardware specialist, it is very dicult to nd which part of the hardware is wrong. The last advantage of this machine is its expansion capability: you can add up to four RAID controller card, the operating system will see only one, facilitating your system expansion. You can also nd expansion case for hard-disks (http://www.amselectronics.com/index-CF2000S.html). FInally, the Escalade RAID controller is fully supported under Linux. Kernel modules are available in Open-Source, and developped by the manufacturer.

7.2 Advices for using such a solution Using RAID volumes is a simple way to build secure le system, but it is a good idea to keep the following direction in mind: • RAID volumes are sensitive to hard-disk failure: RAID level 5 allow the loss of one

hard-disk of the array. Having a spare disk is strongly recommended

• Having a spare disk is not enough, what happenned if you're controller card fail?

You must have a spare RAID controller card. It must be exactly the same, because each card has it own encryption algorythm: so loosing the card is loosing the data.

7.3 RAID level We use RAID 5 level for the following reason: it oers a good compromise between diskspace loss and disk-failure tolerance. For our 12 hard-disk, we only loose the disk-space of one item, while we can accepts the failure of one disk on the array. In an easy way, RAID 5 is based on a ckecksum set out again on every disk.

Page 17

PAIR MeteoSat Second Generation

January 17, 2005

Chapter 8 Network To communicate between each other, the three machines are part of a network. The storage station provides a network accessible volume. This is used by Processing and Reception machine. The size of tranferred data is such, that 100 MBytes minimum is recommended. The ideal way of working is to use 1000 Mbytes networking minimum on the storage machine, and 100 MBytes on the others. A Gigabit Ethernet Switch is now aordable, and the motherboards we use for the storage provides this capability by default. We use a Netgear 5 ports Gigabit Switch: http://www.netgear.fr/produits/switches/gs105.asp

18

Part III Software view of the station:

19

CHAPTER 9.

OVERVIEW

Chapter 9 Overview The goal of the software chain is to transform the raw data segment received from the antenna, to something exploitable for practical experiments. We can see it under dierent view:

9.1 Formats Sequence • .hdr and .wvt: hdr for header: containing ancillary data, wvt is containing the

picture code

• .pgm: lossless pictures format, readable by many pictures viewer. • .pgm.bzip2: result of the compression by bzip2

9.2 Tasks Sequence • tq-tellicast: windows software used to receive picture segments, it also decrypt the

segment using the eToken decryption key. This task is performed on the Receive Station

• move.vbs: visual basic script: perform moving received les to the network volumes • hglue: Linux software ( ported by Jean Claude Bergès), take every segment to

assemble them into one hdr, and one wvt le. This run on the Treatment Station.

• decompwrapper: Linux Software (ported by Julien Cornebise), convert hdr & wvt

les to a pgm pictures les.

• bzip2 compression: Linux Shell Script, compress pgm pictures to free disk-space.

9.3 Logging & locking System We wrote 2 scripts to complete the software chain: move.vbs and Chain.sh. They are both written with the idea that they must care about the order of operations. If an step, for

Page 20

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 9.

OVERVIEW

any reasons, is not performed completely, it is mentionned in a log le. On the Reception station, this log is in D:/, and is named move.log, and reprise.log. When move.vbs is running it creates a lock les in D:/received in order to avoid moving les that don't have to be moved. The system is in place on the Treatment Station: in /services/pgmbz, you can nd when hglue is running the lock le hglue.lock

Page 21

PAIR MeteoSat Second Generation

January 17, 2005

CHAPTER 10.

NETWORK CONFIGURATION

Chapter 10 Network Conguration The network conguration of the station is not very dicult: you must have your three or more machines in a class 3 network. For instance: • IP Adress: Reception (192.168.0.1), Treatment (192.168.0.2), Storage(192.168.0.3) • Netmask: 255.255.255.0

All computer are plugged on the switch. The software chain use Netbios Shared Volumes. That's why you need to congure on the Storage Station Samba shared volumes. You can see in appendices the conguration les we use.

Page 22

PAIR MeteoSat Second Generation

January 17, 2005