[IGSMAIL-1686] New RINEX Compression

[W.]

IGS Electronic Mail      Fri Sep 19  8:20:37 PDT 1997      Message Number 1686
******************************************************************************

Author: W. Gurtner
Subject: New RINEX Compression: Status and Next Steps


          Hatanaka RINEX Compression: Successful First Testphase
          ******************************************************

                             Werner Gurtner
                         Astronomical Institute
                          University of Berne


Since April 1997 the RINEX compression scheme developed by Yuki Hatanaka,
GSI Japan, has been extensively tested at least between the following IGS
Data and Analysis Centers: CDDIS, SIO, IGN, IfAG, CIGNET, GSI, ESA, GRAZ,
and CODE. 

To my knowledge a few other centers and users also took the opportunity to
download the available observation files compressed under the new scheme.

These centers convert all incoming Unix-compressed RINEX observation
files (*.o.Z or *.O_Z) into the compact ones according to the Hatanaka
compression scheme (*.d.Z or *.D_Z) and exchange among themselves these compact
files only. To the users the Unix-compressed files are still made available.

The Hatanaka compression is a two-stage procedure (see IGSMail 1525): It first
compresses the RINEX files by making use of the knowledge of the RINEX format
and contents (by forming third- or fourth-order differences of subsequent
observations) and secondly it uses the Unix-compression to further compress
these intermediate (ASCII) files. It is applied to RINEX observation files
only. There are script (command) files available that distinguish between RINEX
observation files and other files and call just one or both compression (or
decompression) steps accordingly.

The most recent versions (Sept 16, 1997) of the C programs and command files
for all major platforms can be obtained at

  ftp://terras.gsi-mc.go.jp/pub/software/RNXCMP_2.1beta/

or
 
  Internet:   ubeclu.unibe.ch
  User:       anonymous
  Directory:  cd aiub$ftp    (that's a hidden directory)
              cd RNXCMP      (do the change in these two steps!)

The tests showed the following results:

Average disk space (and transmission time) savings:

  Factor of 2.5 compared to the UNIX-compressed-only observation files
  factor of 7.5 compared to the uncompressed RINEX (ASCII) files

Additional computer time used for decompression:

  "Negligible"  (SIO)
  "no problems" (IfAG)
  "3.5 minutes per site on the VAX" (CDDIS)
  "2 minutes per file on the VAXstation 4000/90, a few seconds on the HP
   server" (IGN)
  "A few seconds per file on the DEC VMS Alpha computer" (CODE)

Installation and use of the software:

  "The process of switching  from compressed RINEX to compact/compressed RINEX
   went very well" (CIGNET) 
  "The new compression demonstrated to be less prone to give corrupted
   data because of the smaller size of the files" (ESA)
 
   Serious problems encountered: None


Next Steps:
**********

The following steps have been proposed to the IGS Governing Board and accepted
by the Board at its most recent meeting in Rio de Janeiro in September 1997:


1) End of September 97:

   - Distribute through IGSMail the results of the currently running test phase
     (i.e. this mail)

******************************************************************************
*                                                                            *
*  - Encourage the remaining IGS Centers to join the test phase, i.e.,       *
*    to donwload compact RINEX files where available (Analysis Centers)      *
*    or to generate compact RINEX files for data upload (Operational         *
*    Centers and individual stations), at least for a limited data volume    *
*                                                                            *
*  - Encourage other users of IGS tracking data to download and use compact  *
*    RINEX files where available                                             *
*                                                                            *
******************************************************************************

2) Till end of November 97  

   - Ask for comments (through IGSMail,to be sent out as reminder first half
     of November)

3) Governing Board Meeting in December 1997:

   - Proposal to the IGS Governing Board to switch from the standard
     compressed RINEX to the compact RINEX files, if the extended test phase
     does not reveal major problems nor generate substantial resistance from
     the IGS community

If accepted:

4) Till February 1st, 1998

   - Introduction of the compact RINEX files for data distribution at and
     between all IGS Data Centers (Global and Regional)

5) Till March 1st, 1998

   - Introduction of the compact RINEX files for the data transfer between the
     individual stations or Operational Centers and the assigned IGS Data
     Centers according to bilateral coordination

6) May 1st, 1998

   - Availability of new standard RINEX files at the Data Centers may be
     discontinued

Thanks to all the colleagues who participated in the tests till now and special
thanks to Yuki for the development of the software.

                                                          Werner Gurtner



[Mailed From: Werner Gurtner <GURTNER at aiub.unibe.ch>]