[IGS-DCWG-191] Re: [IGS-IC-816] Re: compression file changes ... [SEC=UNCLASSIFIED]

Wed Jan 9 23:43:20 PST 2019

Thanks Rolf!

The MGEX products are definitely included in the transition to .gz (plus 
with the long product names!) , there is a separate recommendation for 
the MGEX products as we will also move to merge the product structures, 
so we will get to it as well for any specific comments anyone may have...

Best regards,
Nacho
_______________________________________________________________

Ignacio (Nacho) Romero
Aerospace Engineer, PhD
CSC Ltd @ ESA/ESOC/OPS-GN
Satellite Navigation Engineer @ ESOC Navigation Support Office
IGS Infrastructure Committee Chair
www.navigation-office.esa.int
www.canaryspaceconsulting.co.uk
_______________________________________________________________
This message including any attachments may contain confidential
information, according to our Information Security Management System,
  and intended solely for a specific individual to whom they are addressed.
  Any unauthorised copy, disclosure or distribution of this message
  is strictly forbidden. If you have received this transmission in error,
  please notify the sender immediately and delete it.

On 10/01/2019 07:33, Rolf Dach wrote:
> Hi Nacho and colleagues,
>
> from an AC (or user of the DC facilities) perspective, the change from 
> .Z to .gz should not be an issue. We at CODE have changed the 
> compression program already for our internal archive.
>
> Regarding your point 2) I understand the concerns on the DC side to 
> change the files in the archive (what was already a reasonable effort 
> only with our internal archive at CODE). On the other hand, it is from 
> the user perspective from very comfortable if one has to change the 
> search pattern for downloading a file at a certain date and also the 
> internal pre-processing (meaning decompression).
>
> A related question is regarding the products form MGEX with long 
> filenames. Should they also be provided (and submitted?) with gzip (to 
> be consistent there)?
>
> Best regards
>
> Rolf
>
>
>
> Am 10.01.2019 um 04:00 schrieb Ruddick Ryan:
>> Hi Nacho.
>>
>> A bit of background first.
>>
>> At Geoscience Australia we have recently completed a new GNSS data 
>> repository (I will send an email announcing this shortly). Early on 
>> in the development we hit the problem of UNIX compression not being 
>> supported by the modern systems/applications, we spent some time 
>> writing the code in Python to support the recompression but in the 
>> decided against using it. As such we made a decision to transition 
>> all of our historic RINEX 2 data to gzip. We are still submitting 
>> files to IGS data centres using UNIX compression and making a copy of 
>> files available via old FTP (not via our new data repository, which 
>> is not going to support FTP). We do at some point in the near future 
>> plan on only supporting and submitting RINEX 3 files.
>>
>> In answer to your questions.
>>
>> 1.Geoscience Australia is supportive of the change and have no 
>> problem with the proposed date. We are in a position to submit RINEX 
>> files to IGS DC’s in any compression format requested (our preference 
>> is for gzip).
>>
>> 2.On a global data centre level I agree that changing the compression 
>> of historic files could cause problems. That said at a local/regional 
>> level, as stated above we will (have) change(d) all of our historical 
>> files to use gzip compression. To continue supporting legacy 
>> applications/users we will retain a UNIX compressed copy of the files 
>> available only via FTP on our old data centre.
>>
>> 3.I am not aware of any specific place, but there may be commercial 
>> software packages that will need changes made to them. This could 
>> cause problems with legacy software where patching support is no 
>> longer provided.
>>
>> Regards,
>>
>> Ryan
>>
>> UNCLASSIFIED
>>
>> Classified by ryan.ruddick at ga.gov.au on 10/01/2019 2:00:21 PM
>>
>> *From:*IGS-IC <igs-ic-bounces at lists.igs.org> *On Behalf Of *Nacho Romero
>> *Sent:* Tuesday, 8 January 2019 9:28 PM
>> *To:* igs-dcwg at lists.igs.org; ic IGS List <igs-ic at lists.igs.org>; Lou 
>> Estey <lou at unavco.org>; David Maggert <dmaggert at unavco.org>; MacLeod, 
>> Ken <Ken.MacLeod at NRCan-RNCan.gc.ca>
>> *Cc:* gsfc-cddis-ops at lists.nasa.gov; Limbacher, Rebecca Irene. 
>> (GSFC-690.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 
>> <rebecca.i.limbacher at nasa.gov>; mark.van.kints-esa.int 
>> <mark.van.kints at esa.int>
>> *Subject:* [IGS-IC-814] compression file changes ...
>>
>> Dear all,
>>
>> At the last IGS workshop at the IC+DC splinter meeting we approved 
>> the recommendation ...
>>
>> *2018-4. *To coordinate DC, IC and NC so as to move away from Z 
>> compression for all IGS files (data and products) and into using gzip 
>> (.gz)
>>
>> Please provide us comments on the following;
>>
>> 1) I propose that we announce this change now with the date 01 June 
>> 2019 , and at that point we publish all the newly submitted files 
>> (data and products) as .gz in all IGS DCs, whether the owner submits 
>> it that way or not ... do you agree that DCs should enforce such a 
>> change or should we insist all the owners transition their own 
>> submitted files?
>>
>> 2) Do we change all the historical files to .gz ? This will generate 
>> "new files" which the myriad of sync processes (private and public) 
>> running between all sort of servers will cause a lot of duplications 
>> ... I propose we leave the old files as they are for now to avoid a 
>> lot of potential duplication of files in IGS and user's servers
>>
>> 3) are you aware of specific places were the .Z compression is 
>> requested? required? recommended? ... I think the site guidelines are 
>> generic enough, what do you think?
>>
>> We have some other recommendations to discuss with you in the next 
>> few days, and in the next months I will organize a telecon so we can 
>> all agree a way forward for each one.
>>
>> Please let us know your comments on this specific recommendation and 
>> I will collate all the discussion points for the coordination 
>> telecon. Thanks.
>>
>> -- 
>>
>> Best regards,
>>
>> Nacho
>>
>> _______________________________________________________________
>>
>> Ignacio (Nacho) Romero
>>
>> Aerospace Engineer, PhD
>>
>> CSC Ltd @ ESA/ESOC/OPS-GN
>>
>> Satellite Navigation Engineer @ ESOC Navigation Support Office
>>
>> IGS Infrastructure Committee Chair
>>
>> www.navigation-office.esa.int <http://www.navigation-office.esa.int>
>>
>> www.canaryspaceconsulting.co.uk <http://www.canaryspaceconsulting.co.uk>
>>
>> _______________________________________________________________
>>
>> This message including any attachments may contain confidential
>>
>> information, according to our Information Security Management System,
>>
>>   and intended solely for a specific individual to whom they are 
>> addressed.
>>
>>   Any unauthorised copy, disclosure or distribution of this message
>>
>>   is strictly forbidden. If you have received this transmission in 
>> error,
>>
>>   please notify the sender immediately and delete it.
>>
>> Geoscience Australia Disclaimer: This e-mail (and files transmitted 
>> with it) is intended only for the person or entity to which it is 
>> addressed. If you are not the intended recipient, then you have 
>> received this e-mail by mistake and any use, dissemination, 
>> forwarding, printing or copying of this e-mail and its file 
>> attachments is prohibited. The security of emails transmitted cannot 
>> be guaranteed; by forwarding or replying to this email, you 
>> acknowledge and accept these risks.
>> ------------------------------------------------------------------------------------------------------------------------- 
>>
>>
>>
>> _______________________________________________
>> IGS-IC mailing list
>> IGS-IC at lists.igs.org
>> https://lists.igs.org/mailman/listinfo/igs-ic
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20190110/47a7d8b2/attachment.html>