[IGS-DCWG-184] Re: highrates tar file
Nacho Romero
nacho at canaryspaceconsulting.co.uk
Mon Nov 12 06:25:49 PST 2018
Thanks Erwin,
If agreed we will place the tar files in the highrate directories were
they are now ... its a very small change actually just eliminating the
hourly directories and making 1 tar file from the 96 ... in terms of the
Rinex 3 tar file naming I am not sure what Carey had in mind. We had the
idea for the Rinex 2 to use the AAAADDD0.18T.tar naming (TBC) and I
thought we could use ".tar" as the compression method as you propose
"ABMF00GLP_R_20183110000_01D_01S_MO.crx.tar" ... this is perfectly
in-line with the Rinex 3 data format document, and inside we place the
96 15M_01S.crx.gz files ... is that deemed confusing ? incorrect?
Using gfzrnx or teqc are not perfect alternatives as we have seen issues
over time with both tools and the headers are changed when merging 96
headers into 1 plus they could reorder observables and constellations,
change Rinex version numbers, etc... and if there are gaps (missing
15min files) we will lose all traceability as to where the gap happened
and who may have introduced a problem ... . I still would favor a tar of
the existing files, it is cleaner and much more obvious. But if the DC
WG or the EPN feels differently or do not feel like a change is needed
please let us know and we can easily table this recommendation for later
consideration, no problem. Also if the EPN do not agree with this IGS
recommendation at all and have their own way forward we can find a
common solution , at the workshop no objections were raised. But raising
them now is fine of course.
Many operators (like ESA) actually merge at the raw binary level sbf ,
tps, jps, etc files and then generate hourly and daily files over the
day and we have seen content differences to files merged at RINEX level
using external tools and differences to stream files of course. The
manufacturer tools from the raw binaries is what many of us use to
generate Rinex files so there is no "one solution". In any case the
source of the files should be as much as possible the Station Operators
. The DCs should be as much as possible preserving what the owners of
the data give us and not inventing things for the users ... its easy to
get hourly files for processing we all do it for the ultra-rapids ...
If you are not keen on merging data files stop doing it and demand the
SOs do their own work for their own station's data. Making them
responsible is actually what they signed up for. It is clear in the IGS
at least that every station agrees to submit as a minimum Daily data
files (see IGS site guidelines) and then optionally hourly, 15min and
data streams but whatever they decide they are responsible and we will
check they are living to their commitments, this way the DCs are not
loaded with extra responsibilities. But its just a thought of course.
Mixing 01S and 30S data in the same directory has never seemed like a
good idea, nor mixing hourly and daily files in the same directory, not
sure why that would be a good idea now. Even if Rinex 3 and Rinex 2
supports it having thousands of files together makes things potentially
slower why propose that now? Is there a benefit to the flatter
structure possible? why even have day directories? ;) I guess if all
file accesses are symbolic it would just be a convention and we could
access each file via many different routes ... regional directories,
station directories, etc in the end each symbolic link in a drive is not
unique! but the ftp way of thinking has us a bit constrained that is true!
Regards,
Nacho
On 12/11/2018 13:09, Erwin.Wiesensarter at bkg.bund.de wrote:
>
> Nacho,
>
> I’m not keen on merging files but it’s simply a fact that not all
> stations provide daily files. Some create hourly files only and ask us
> to merge them. So what is the alternative? Having no data at all. Great.
>
> Beside that operators usually would use the same “standard” tools as
> we do, teqc or gfzrnx, and it’s even commented in the header.
>
> For RINEX3 files you would get a file like
> ABMF00GLP_R_20183110000_01D_01S_MO.crx.gz. The name is clear for
> everyone, thanks to the new naming convention. It could be put without
> any trouble in the same directory as
> ABMF00GLP_R_20183110000_01D_30S_MO.crx.gz.
>
> But ok, as everyone prefers tar files,
>
> -what is the proposed filename:
> ABMF00GLP_R_20183110000_01D_01S_MO.tar? containing gzip compressed files
>
> -Where to put them, to daily directories beside 30sec files?
>
> Regards,
>
> Erwin
>
> *Von:*Nacho Romero [mailto:nacho at canaryspaceconsulting.co.uk]
> *Gesendet:* Montag, 12. November 2018 13:15
> *An:* Wiesensarter, Erwin <Erwin.Wiesensarter at bkg.bund.de>; Noll,
> Carey <Carey.E.Noll at nasa.gov>
> *Cc:* Goltz, Markus <Markus.Goltz at bkg.bund.de>
> *Betreff:* Re: AW: highrates tar file
>
> Hi ,
>
> and the IGS is precisely against this practice! as stated many, many
> times we do not agree with DCs creating Rinex files except 15min 1 sec
> files from streams, all other files in our estimation should come from
> station operators . We have had this discussion many times over the
> years especially when BKG thought they could do this to ESA sites when
> our hourly or daily files were delivered late creating A LOT of
> confusion to our users. DCs are in general NOT allowed to create 30s
> files unless given the explicit permission by the station operator. I
> do not think that a data center (which is like a library) should be
> creating new books by concatenating the works of each author ... its a
> fundamental issue in how you interpret the DCs role. I know you
> disagree that is fine but for IGS the line is very clear.
>
> besides for Rinex 2 we would have to invent a whole new Rinex data
> filename since 24H_01S files have never been specified in Rinex 2
>
> Where would the confusion come? what is inconsistent? We are precisely
> retaining all the consistency from the SO or stream files and just
> removing a bunch of hourly subdirectories and making 1 file to
> download instead of 96!
>
> Please let me know what "users" will be confused or is it just that
> you prefer to create a 24H file? and in that case what tool ? and what
> if there is a problem with the tool? or different DCs use different
> tools creating different 24H_01S files? its just much cleaner to
> create a tar of the files we have and not mess with the data from the
> station operators.
>
> The DCs are the "keepers" of the SOs data we should respect what they
> submit and make it easier for users to find what the SOs submitted but
> its not the DCs job to "serve" the users but rather store the
> submitted data with as much fidelity as possible .
>
> regards,
>
> Nacho
>
> On 12/11/2018 12:02, Erwin.Wiesensarter at bkg.bund.de
> <mailto:Erwin.Wiesensarter at bkg.bund.de>wrote:
>
> Hi Nacho,
>
> we are already doing this when we (DC) merge hourly files to daily
> ones. Not all stations send daily files.
>
> That’s a bit the inconsistency and confusion for the users, that
> bothers me.
>
> Erwin
>
> *Von:*Nacho Romero [mailto:nacho at canaryspaceconsulting.co.uk]
> *Gesendet:* Montag, 12. November 2018 12:57
> *An:* Wiesensarter, Erwin <Erwin.Wiesensarter at bkg.bund.de>
> <mailto:Erwin.Wiesensarter at bkg.bund.de>; Noll, Carey
> <Carey.E.Noll at nasa.gov> <mailto:Carey.E.Noll at nasa.gov>
> *Cc:* Goltz, Markus <Markus.Goltz at bkg.bund.de>
> <mailto:Markus.Goltz at bkg.bund.de>
> *Betreff:* Re: highrates tar file
>
> Hi Erwin,
>
> Yes, there is reason, since Data Centers should in our estimation
> not create new data files. We think that consolidation into one
> tar file makes it easier to download the 1sec data while retaining
> the original header/observables as intended by the station
> operators . Also if researchers already handle 15M_01S files then
> the tar is a small step whereas a new 24H_01S (or 01D_01S) is a
> completely new data product not specified so far in the IGS
>
> Sincerely,
>
> Nacho
>
> On 12/11/2018 11:51, Erwin.Wiesensarter at bkg.bund.de
> <mailto:Erwin.Wiesensarter at bkg.bund.de>wrote:
>
> Dear Carey, Nacho,
>
> Is there a special reason why you promote a "daily" tar-file -
> a new product - instead of a concatenated regular daily file
> with *_01S_* sampling rate?
>
> Regards,
>
> Erwin
>
> --
>
> Best regards,
>
> Nacho
>
> _______________________________________________________________
>
> Ignacio (Nacho) Romero
>
> Aerospace Engineer, PhD
>
> CSC Ltd @ ESA/ESOC/OPS-GN
>
> Satellite Navigation Engineer @ ESOC Navigation Support Office
>
> IGS Infrastructure Committee Chair
>
> www.navigation-office.esa.int <http://www.navigation-office.esa.int>
>
> www.canaryspaceconsulting.co.uk
> <http://www.canaryspaceconsulting.co.uk>
>
> _______________________________________________________________
>
> This message including any attachments may contain confidential
>
> information, according to our Information Security Management System,
>
> and intended solely for a specific individual to whom they are
> addressed.
>
> Any unauthorised copy, disclosure or distribution of this message
>
> is strictly forbidden. If you have received this transmission in
> error,
>
> please notify the sender immediately and delete it.
>
> --
> Best regards,
> Nacho
> _______________________________________________________________
> Ignacio (Nacho) Romero
> Aerospace Engineer, PhD
> CSC Ltd @ ESA/ESOC/OPS-GN
> Satellite Navigation Engineer @ ESOC Navigation Support Office
> IGS Infrastructure Committee Chair
> www.navigation-office.esa.int <http://www.navigation-office.esa.int>
> www.canaryspaceconsulting.co.uk <http://www.canaryspaceconsulting.co.uk>
> _______________________________________________________________
> This message including any attachments may contain confidential
> information, according to our Information Security Management System,
> and intended solely for a specific individual to whom they are addressed.
> Any unauthorised copy, disclosure or distribution of this message
> is strictly forbidden. If you have received this transmission in error,
> please notify the sender immediately and delete it.
> INVALID HTML
--
Best regards,
Nacho
_______________________________________________________________
Ignacio (Nacho) Romero
Aerospace Engineer, PhD
CSC Ltd @ ESA/ESOC/OPS-GN
Satellite Navigation Engineer @ ESOC Navigation Support Office
IGS Infrastructure Committee Chair
www.navigation-office.esa.int
www.canaryspaceconsulting.co.uk
_______________________________________________________________
This message including any attachments may contain confidential
information, according to our Information Security Management System,
and intended solely for a specific individual to whom they are addressed.
Any unauthorised copy, disclosure or distribution of this message
is strictly forbidden. If you have received this transmission in error,
please notify the sender immediately and delete it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20181112/f11d1091/attachment-0001.html>
More information about the IGS-DCWG
mailing list