[IGS-DCWG-11] Re: [IGS-DCWG-7] Resupplied data

[Edouard GAULUE]

******************************************************************************
IGS-DCWG Mail      16 Jun 09:01:40 PDT 2003      Message Number 11
******************************************************************************

Author: Edouard Gaulue

Dear Colleagues,

The problem really exist.

Preparing the Ottawa meeting, we tried to think about it=20
with Loic.

We discuss two points of view:
- To try to adapt the rcs/cvs logic and tools. The problem=20
is how to deal with it. It musn't be centralized and so on.=20
Anyway there is certainly something to find looking in this=20
direction. But this mean finding time or people to work on it.
- To create a very simple system based on present ressources=20
totaly distributed that would allow statistic making on=20
centers and network. Here is the idea:

When a file is created and is good for distribution in an=20
operationnal center another file (like navigation "n" or=20
quality check "s") should be created with:

 > Submit by $operational_center at $timestamp ($file_size)

When received by a data center (regional or global), this=20
file as to be appended:

 > Received by $data_center from $other_data_center at=20
$timestamp ($file_size)

And so on...

Different GDC can have different "nsq" (for network service=20
quality) files. But in each center, this file reflect the=20
history of the data file associated.

In case of resubmission the operationnal center append its=20
own file:

 > Resubmit by $operational_center at $new_timestamp=20
($new_file_size)

When we compare this file with our we can easily guess a=20
change has occured and get the new data file. So we restart=20
the procedure with this new "nsq" source file.

Here is an exemple of a resubmited file we could find at ign=20
center:

File name: bzrg1610.03d.Z.nsq associated with bzrg1610.03d.Z

Submit by ASI at 160-03/03:10:27 (331234)
Resubmit by ASI at 164-03/10:10:27 (331259)
Received by BKG from ASI at 164-03/13:45:24 (331259)
Received by IGN from BKG at 164-03/15:30:59 (331259)


In this manner:
- at all data center stages everyone is able to know the=20
history of the data file associated (just need FTP)
- also work with navigation, meteo and so on ...
- in case of desagreement between nsq file and associated=20
data file (based on size) it's easy to send an automatic email
- if operationnal center can't (or don't want to) provide=20
this service, the above data center can simulate the=20
operationnal data
- we can make statistics of file arrival in each center and=20
understanding our network behavior
- we can decide one or two centers always look routinely to=20
those source files in order to make supervision (writing=20
automatic IGSMail, buiding a request service able to answer=20
speedly last size and last timestamp of a file, ...)

Of course, it would be even greater if those files were XML=20
structured.

I think it's very easy to write few beginning rules (one=20
diffulcty is to agree on the timestamp) and to implement them.

Here are just few ideas. Please feel free to react.

EG


Carey Noll wrote:
> ***********************************************************************=
*******
> IGS-DCWG Mail      04 Jun 06:24:52 PDT 2003      Message Number 7
> ***********************************************************************=
*******
>=20
> Author:  Carey Noll/CDDIS
>=20
> Dear Colleagues,
>=20
>    I have received several emails about how resubmitted data are handle=
d.
> I would like to get some ideas from the working group on how the IGS co=
uld
> better handle re-posting of data sets.  I know that at the CDDIS, for=20
> example, we get any resupplied data manually where messages are sent ou=
t
> through IGSMail.  However, I also know that we often get a second versi=
on
> of files automatically delivered to our incoming data areas.  Do any of=

> you have ideas as to how we can better handle re-posting of data to all=
 data=20
> centers?  Are any of you using mirroring s/w routinely and would
> this help?  How about user community notification?  Should we set up a
> website to automatically post information about resupplies or an email
> notification list?  I am just throwing some ideas out here for discussi=
on. =20
> Perhaps I am worried about a problem that doesn't exist?
>=20
> Regards,
> Carey.
>=20
> +=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D=
+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=

> Ms. Carey Noll  Manager, Crustal Dynamics Data Information System (CDDI=
S)
>                                            Secretary, ILRS Central Bure=
au
> Code 922                                   E-mail:  Carey.E.Noll at nasa.g=
ov
> NASA GSFC                                           Voice: (301) 614-65=
42
> Greenbelt, MD 20771                                   Fax: (301) 614-60=
99
> USA                  WWW:  http://cddisa.gsfc.nasa.gov/cddis_welcome.ht=
ml
> +=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D=
+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=3D+=

>=20


--=20
Edouard GAULUE
SGN - IGN - 2 avenue Pasteur - 94160 Saint Mand=E9 - France
Tel : +33 (0) 1 64 15 32 43
Mail : Edouard.Gaulue at ensg.ign.fr