[IGS-DCWG-62] RE: [IGS-DCWG-60] Re: [IGS-DCWG-59] Re: Change in compression scheme at IGS data centers
Schmidt, Michael
Michael.Schmidt at NRCan-RNCan.gc.ca
Wed Mar 31 10:31:20 PDT 2010
******************************************************************************
IGS-DCWG Mail 31 Mar 10:31:26 PDT 2010 Message Number 62
******************************************************************************
Author: Mike Schmidt
Very Spooky - Trust Angie's excellent memory to recall old email threads
- ... LOL -
Not sure what my-future-self thinks, but would put the following forward
for contemplation:
As we all recognize, the DC's serve both the IGS community (i.e. mainly
UNIX, Linux, OS-X, etc. users, command line scripting, etc.) as well as
the non-IGS community (i.e. mainly PC/Windows users). For example many
vendors' software access IGS data centers auto-magically and retrieve
the closest reference station(s) for their processing - presumably their
software handles the de-compression. There are also many (PC) users who
download data manually.
The basic criteria might therefore be:
- our compression scheme(s) must be supported by:
- popular PC based decompression software (e.g. WinZip, etc.);
- multi-platforms;
- vendors' software;
- other (??)
- compression software has to be useable in a scriptable environment
(UNIX, LINUX, OS-X, DOS ...);
- guaranteed long term developer support for the compression software;
- other (??)
If bzip2 meets the criteria above and provides a demonstrable saving
then it could be considered. As Dave noted, the change would basically
be a script change so fairly straight forward and the timing is good.
The other issue we should consider refers to the Hatanaka compression:
- What is the future of the Hatanaka compression scheme?
- Is there any implication wrt addition of new GNSS signals on the
Hatanaka compression?
- Will Hatanaka compression software be supported and/or open-sourced so
that we can continue to rely upon this as part of the long-term IGS
compression scheme?
And, as Carrie notes, if we are going to change, a well advertised
transition period is essential.
Cheers from my-future-self
Mike
-----Original Message-----
From: owner-igs-dcwg at igscb.jpl.nasa.gov
[mailto:owner-igs-dcwg at igscb.jpl.nasa.gov] On Behalf Of Stowers, David A
(3350)
Sent: Tuesday, March 30, 2010 11:49 AM
To: igs-dcwg at igscb.jpl.nasa.gov
Subject: [IGS-DCWG-60] Re: [IGS-DCWG-59] Re: Change in compression
scheme at IGS data centers
************************************************************************
******
IGS-DCWG Mail 30 Mar 11:49:38 PDT 2010 Message Number 60
************************************************************************
******
Author: Dave Stowers
Spooky :)
One time hit to change data handling scripts, but probably
a reasonable shift long-term. We're moving more to redundant hard disk
based storage vs. "burned" optical
media, so it's just processing cycles and customer notification on
this end.
-dave
On Mar 30, 2010, at 11:35 AM, "Angelyn W. Moore"
<Angelyn.W.Moore at jpl.nasa.gov
> wrote:
> ***
> ***
> ***
> *********************************************************************
> IGS-DCWG Mail 30 Mar 11:35:51 PDT 2010 Message Number 59
> ***
> ***
> ***
> *********************************************************************
>
> Author: Angelyn Moore
>
> I have been sent 5 years into the future to agree with this
> suggestion (see
> below).
>
> What does 2005-Mike-Schmidt's future self think?
>
> A transition period would be in order, to prepare software and
> educate non-IGS
> users (so as to end up wanting to travel into the past to undo the
> decision).
>
> Angie
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Angelyn W. Moore, Ph.D. Geodynamics and Space Geodesy Group,
> 335N
> JPL/Caltech
> Angelyn.W.Moore at jpl.nasa.gov
> 4800 Oak Grove Dr. MS 238-540 voice: +1 818 354
> 5434
> Pasadena CA 91109 USA fax: +1 818 393
> 6686
>
>
> Return-path: <Angelyn.W.Moore at jpl.jpl.nasa.gov>
> Date: Tue, 22 Mar 2005 10:26:38 -0800
> From: "Angelyn W. Moore" <Angelyn.W.Moore at jpl.nasa.gov>
> Subject: Re: CB www server
> In-reply-to:
<7CB5F45D9E56D511836F0002A537DD6992C8CC at s5-sid-r1.nrcan.gc.ca
> >
> To: "Schmidt, Michael" <MSchmidt at NRCan.gc.ca>
> Cc: Angelyn W Moore <Angelyn.W.Moore at jpl.nasa.gov>
>
>> Wrt earlier discussions on compression schemes:
>> (a) has this gone anywhere?
>
> Not that I know of :)
>
>> (b) are you familiar with bzip2? If so any comments on this?
>
> Have used it with some frequency under linux, albeit not for
> GPS data or prods. No adverse experiences. Works.
> Cheers
> Angie
> ~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Angelyn W. Moore, Ph.D. Deputy Director, IGS Central
> Bureau
> JPL/Caltech
Angelyn.W.Moore at jpl.nasa.gov
> 4800 Oak Grove Dr. MS 238-540 voice: +1 818 354
5434
> Pasadena CA 91109 USA http://igscb.jpl.nasa.gov fax: +1 818 393
6686
************************************************************************
******
IGS-DCWG Mail 30 Mar 11:23:01 PDT 2010 Message Number 58
************************************************************************
******
Author: Carey Noll/CDDIS
Over the years we have discussed updating the compression scheme
used within the IGS to a more modern utility. I would once again like to
visit this topic. Of course any change such as this would have to be
rolled out over time but, at the CDDIS at least, we are thinking about
changing the compression used in older data sets and then migrating more
recent data over time, closely coordinating with our various user
communities. Any new method should be integrated (or freely available)
in the OS of major platforms (Linux, UNIX, Windows, OS X). Staff here
are recommending bzip. This utility is available on Linux, UNIX, and OS
X releases and is available for free for Windows. We have seen an
improvement in compression by 20-30% using bzip on our limited tests. I
would be interested in any thoughts on this proposal as well as any
other commonly available utilities that could be considered.
Regards,
Carey.
-----
Ms. Carey Noll
Manager, Crustal Dynamics Data Information System (CDDIS) Secretary,
ILRS Central Bureau NASA GSFC Code 690.1 Greenbelt, MD 20771 USA
E-mail: Carey.Noll at nasa.gov
Voice: (301) 614-6542
Fax: (301) 614-6015
WWW: http://cddis.gsfc.nasa.gov
More information about the IGS-DCWG
mailing list