[IGS-DCWG-186] Re: [IGS-IC-810] highrates tar file
Noll, Carey E. (GSFC-61A0)
carey.e.noll at nasa.gov
Tue Nov 13 12:03:58 PST 2018
Hi Nacho,
Just returning from the ILRS workshop and digesting all my emails. My quick thoughts were as you detailed below, one tar file/site/type/version/day, such as:
gode0010.18d.tar for all GODE high-rate compact obs files for day 2018001
GOP700CZE_R_20180010000_01D_01S_M).crx.tar for all GOP7 high-rate compact obs files for day 2018001
Similar for other types (nav, etc.). And I would propose these files be archived in:
/gnss/data/highrate/YYYY/DDD/YYT
Does that work?
Carey.
-----
> On Nov 12, 2018, at 9:25 AM, Nacho Romero <nacho at canaryspaceconsulting.co.uk> wrote:
>
> Thanks Erwin,
>
> If agreed we will place the tar files in the highrate directories were they are now ... its a very small change actually just eliminating the hourly directories and making 1 tar file from the 96 ... in terms of the Rinex 3 tar file naming I am not sure what Carey had in mind. We had the idea for the Rinex 2 to use the AAAADDD0.18T.tar naming (TBC) and I thought we could use ".tar" as the compression method as you propose "ABMF00GLP_R_20183110000_01D_01S_MO.crx.tar" ... this is perfectly in-line with the Rinex 3 data format document, and inside we place the 96 15M_01S.crx.gz files ... is that deemed confusing ? incorrect?
>
> Using gfzrnx or teqc are not perfect alternatives as we have seen issues over time with both tools and the headers are changed when merging 96 headers into 1 plus they could reorder observables and constellations, change Rinex version numbers, etc... and if there are gaps (missing 15min files) we will lose all traceability as to where the gap happened and who may have introduced a problem ... . I still would favor a tar of the existing files, it is cleaner and much more obvious. But if the DC WG or the EPN feels differently or do not feel like a change is needed please let us know and we can easily table this recommendation for later consideration, no problem. Also if the EPN do not agree with this IGS recommendation at all and have their own way forward we can find a common solution , at the workshop no objections were raised. But raising them now is fine of course.
>
> Many operators (like ESA) actually merge at the raw binary level sbf , tps, jps, etc files and then generate hourly and daily files over the day and we have seen content differences to files merged at RINEX level using external tools and differences to stream files of course. The manufacturer tools from the raw binaries is what many of us use to generate Rinex files so there is no "one solution". In any case the source of the files should be as much as possible the Station Operators . The DCs should be as much as possible preserving what the owners of the data give us and not inventing things for the users ... its easy to get hourly files for processing we all do it for the ultra-rapids ...
>
> If you are not keen on merging data files stop doing it and demand the SOs do their own work for their own station's data. Making them responsible is actually what they signed up for. It is clear in the IGS at least that every station agrees to submit as a minimum Daily data files (see IGS site guidelines) and then optionally hourly, 15min and data streams but whatever they decide they are responsible and we will check they are living to their commitments, this way the DCs are not loaded with extra responsibilities. But its just a thought of course.
>
> Mixing 01S and 30S data in the same directory has never seemed like a good idea, nor mixing hourly and daily files in the same directory, not sure why that would be a good idea now. Even if Rinex 3 and Rinex 2 supports it having thousands of files together makes things potentially slower why propose that now? Is there a benefit to the flatter structure possible? why even have day directories? ;) I guess if all file accesses are symbolic it would just be a convention and we could access each file via many different routes ... regional directories, station directories, etc in the end each symbolic link in a drive is not unique! but the ftp way of thinking has us a bit constrained that is true!
>
> Regards,
>
> Nacho
>
> On 12/11/2018 13:09, Erwin.Wiesensarter at bkg.bund.de wrote:
>> Nacho,
>>
>> I’m not keen on merging files but it’s simply a fact that not all stations provide daily files. Some create hourly files only and ask us to merge them. So what is the alternative? Having no data at all. Great.
>> Beside that operators usually would use the same “standard” tools as we do, teqc or gfzrnx, and it’s even commented in the header.
>>
>> For RINEX3 files you would get a file like ABMF00GLP_R_20183110000_01D_01S_MO.crx.gz. The name is clear for everyone, thanks to the new naming convention. It could be put without any trouble in the same directory as ABMF00GLP_R_20183110000_01D_30S_MO.crx.gz.
>>
>> But ok, as everyone prefers tar files,
>> - what is the proposed filename: ABMF00GLP_R_20183110000_01D_01S_MO.tar? containing gzip compressed files
>> - Where to put them, to daily directories beside 30sec files?
>>
>> Regards,
>> Erwin
>>
>> Von: Nacho Romero [mailto:nacho at canaryspaceconsulting.co.uk]
>> Gesendet: Montag, 12. November 2018 13:15
>> An: Wiesensarter, Erwin <Erwin.Wiesensarter at bkg.bund.de>; Noll, Carey <Carey.E.Noll at nasa.gov>
>> Cc: Goltz, Markus <Markus.Goltz at bkg.bund.de>
>> Betreff: Re: AW: highrates tar file
>>
>> Hi ,
>>
>> and the IGS is precisely against this practice! as stated many, many times we do not agree with DCs creating Rinex files except 15min 1 sec files from streams, all other files in our estimation should come from station operators . We have had this discussion many times over the years especially when BKG thought they could do this to ESA sites when our hourly or daily files were delivered late creating A LOT of confusion to our users. DCs are in general NOT allowed to create 30s files unless given the explicit permission by the station operator. I do not think that a data center (which is like a library) should be creating new books by concatenating the works of each author ... its a fundamental issue in how you interpret the DCs role. I know you disagree that is fine but for IGS the line is very clear.
>>
>> besides for Rinex 2 we would have to invent a whole new Rinex data filename since 24H_01S files have never been specified in Rinex 2
>>
>> Where would the confusion come? what is inconsistent? We are precisely retaining all the consistency from the SO or stream files and just removing a bunch of hourly subdirectories and making 1 file to download instead of 96!
>>
>> Please let me know what "users" will be confused or is it just that you prefer to create a 24H file? and in that case what tool ? and what if there is a problem with the tool? or different DCs use different tools creating different 24H_01S files? its just much cleaner to create a tar of the files we have and not mess with the data from the station operators.
>>
>> The DCs are the "keepers" of the SOs data we should respect what they submit and make it easier for users to find what the SOs submitted but its not the DCs job to "serve" the users but rather store the submitted data with as much fidelity as possible .
>>
>> regards,
>>
>> Nacho
>>
>>
>>
>> On 12/11/2018 12:02, Erwin.Wiesensarter at bkg.bund.de wrote:
>> Hi Nacho,
>>
>> we are already doing this when we (DC) merge hourly files to daily ones. Not all stations send daily files.
>> That’s a bit the inconsistency and confusion for the users, that bothers me.
>>
>> Erwin
>>
>> Von: Nacho Romero [mailto:nacho at canaryspaceconsulting.co.uk]
>> Gesendet: Montag, 12. November 2018 12:57
>> An: Wiesensarter, Erwin <Erwin.Wiesensarter at bkg.bund.de>; Noll, Carey <Carey.E.Noll at nasa.gov>
>> Cc: Goltz, Markus <Markus.Goltz at bkg.bund.de>
>> Betreff: Re: highrates tar file
>>
>> Hi Erwin,
>>
>> Yes, there is reason, since Data Centers should in our estimation not create new data files. We think that consolidation into one tar file makes it easier to download the 1sec data while retaining the original header/observables as intended by the station operators . Also if researchers already handle 15M_01S files then the tar is a small step whereas a new 24H_01S (or 01D_01S) is a completely new data product not specified so far in the IGS
>>
>> Sincerely,
>>
>> Nacho
>>
>> On 12/11/2018 11:51, Erwin.Wiesensarter at bkg.bund.de wrote:
>> Dear Carey, Nacho,
>>
>> Is there a special reason why you promote a "daily" tar-file - a new product - instead of a concatenated regular daily file with *_01S_* sampling rate?
>>
>> Regards,
>> Erwin
>>
>>
>> --
>> Best regards,
>> Nacho
>> _______________________________________________________________
>>
>> Ignacio (Nacho) Romero
>> Aerospace Engineer, PhD
>> CSC Ltd @ ESA/ESOC/OPS-GN
>> Satellite Navigation Engineer @ ESOC Navigation Support Office
>> IGS Infrastructure Committee Chair
>> www.navigation-office.esa.int
>> www.canaryspaceconsulting.co.uk
>> _______________________________________________________________
>> This message including any attachments may contain confidential
>> information, according to our Information Security Management System,
>> and intended solely for a specific individual to whom they are addressed.
>> Any unauthorised copy, disclosure or distribution of this message
>> is strictly forbidden. If you have received this transmission in error,
>> please notify the sender immediately and delete it.
>> --
>> Best regards,
>> Nacho
>> _______________________________________________________________
>>
>> Ignacio (Nacho) Romero
>> Aerospace Engineer, PhD
>> CSC Ltd @ ESA/ESOC/OPS-GN
>> Satellite Navigation Engineer @ ESOC Navigation Support Office
>> IGS Infrastructure Committee Chair
>> www.navigation-office.esa.int
>> www.canaryspaceconsulting.co.uk
>> _______________________________________________________________
>> This message including any attachments may contain confidential
>> information, according to our Information Security Management System,
>> and intended solely for a specific individual to whom they are addressed.
>> Any unauthorised copy, disclosure or distribution of this message
>> is strictly forbidden. If you have received this transmission in error,
>> please notify the sender immediately and delete it.
>> INVALID HTML
> --
> Best regards,
> Nacho
> _______________________________________________________________
>
> Ignacio (Nacho) Romero
> Aerospace Engineer, PhD
> CSC Ltd @ ESA/ESOC/OPS-GN
> Satellite Navigation Engineer @ ESOC Navigation Support Office
> IGS Infrastructure Committee Chair
>
> www.navigation-office.esa.int
> www.canaryspaceconsulting.co.uk
>
> _______________________________________________________________
> This message including any attachments may contain confidential
> information, according to our Information Security Management System,
> and intended solely for a specific individual to whom they are addressed.
> Any unauthorised copy, disclosure or distribution of this message
> is strictly forbidden. If you have received this transmission in error,
> please notify the sender immediately and delete it.
>
> _______________________________________________
> IGS-IC mailing list
> IGS-IC at lists.igs.org
> https://lists.igs.org/mailman/listinfo/igs-ic
-----
Ms. Carey Noll
Manager, Crustal Dynamics Data Information System (CDDIS)
Secretary, ILRS Central Bureau
Secretary, GGOS Bureau for Networks and Observations
NASA GSFC
Code 61A
Greenbelt, MD 20771
USA
E-mail: Carey.Noll at nasa.gov
Voice: (301) 614-6542
Fax: (301) 614-6015
WWW: https://cddis.nasa.gov
More information about the IGS-DCWG
mailing list