[IGS-DCWG-210] Re: [EXTERNAL] [IGS-IC-847] Re: Infrastructure Committee business [SEC=OFFICIAL]

Mon Nov 18 07:09:25 PST 2019

Ryan,

  *   Is the primary driver for this proposal to reduce the total number of individual files being stored in the CDDIS data centre or has there been requests from the user community?

Yes, the main driver is technical.  The number of files for us continues to skyrocket yearly.  This negatively affects both distribution and in house actions for CDDIS.  The underlying issue is the small file problem.  Not going to go into details but any data center including yours is/will be affected.  It simply has to do with the total number of operations that must be conducted for EVERY read or write operation of a file multiplied over millions of files.  Even putting SSD into the mix only serves to delay the issue.  You (GA) somewhat mitigate the problem by combining requests for multiple files within a zip file but you still have to deal with the file access latencies for each of all those small files.

  *   Within our data centre we find that the majority of requests for high-rate data files is for a couple of hours rather than an entire day.

That is similar to what we see but its’ usually centered around the most recent data (less than 6 months of age).  Therefore, the reason we proposed not changing anything for data that was less than 6 months of age.

  *   We see no problems with the proposed name or folder structure. Although as a note should monument name read as station name?

Yes, monument name as station name.

  *   We would support CDDIS (or other GDCs) in making such a change, but would request that this not become a mandatory change.

We weren’t proposing making it mandatory but what we (CDDIS) were going to go forward with.

I would suggest delaying the change until after 1 Jan as support during this period is generally limited due to Christmas and New Year. I would suggest 1 Feb be a more appropriate date. Noting that the impact of these change on users would be minimal.

We can probably live with early next year to implement.

R
Pat

From: IGS-IC <igs-ic-bounces at lists.igs.org> on behalf of Ruddick Ryan <Ryan.Ruddick at ga.gov.au>
Date: Thursday, November 14, 2019 at 8:16 PM
To: Nacho Romero <nacho at canaryspaceconsulting.co.uk>, ic IGS List <igs-ic at lists.igs.org>, "igs-dcwg at lists.igs.org" <igs-dcwg at lists.igs.org>
Subject: [EXTERNAL] [IGS-IC-847] Re: Infrastructure Committee business [SEC=OFFICIAL]

Hi Nacho, Carey and Michael.

Thank you for the proposal addressing recommendation 2018-02.

The team at GA have reviewed the document and have the following points:

  *   Is the primary driver for this proposal to reduce the total number of individual files being stored in the CDDIS data centre or has there been requests from the user community?

  *   Within our data centre we find that the majority of requests for high-rate data files is for a couple of hours rather than an entire day.

  *   Within our data centre we do not store files within a directory structure, but essentially a large bucket that is referenced to a database or index. As such, this proposal does not fit within our design and would not provide us any benefit. We are not constrained by the number of files that we can store and users wanting this data can request multiple files at once which are automatically zipped together on download.

  *   We see no problems with the proposed name or folder structure. Although as a note should monument name read as station name?

  *   We support the consolidation of multi-GNSS navigation files into a single directory.

  *   We agree that keeping the most recent set of data in the standard format is a good idea.

  *   We would support CDDIS (or other GDCs) in making such a change, but would request that this not become a mandatory change.

Kind regards,

Ryan Ruddick
Section Leader | GNSS Infrastructure and Informatics
National Positioning Infrastructure | Positioning and Community Safety Division

t +61 2 6249 9426    m +61 (0) 429 771 069
[16-9481 GA Email Signature_social media-04]<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ga.gov.au_&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=DQQWSvdvuxCcNrWfbGGQ9pm4FZo5EzSIcARM0lgvHXU&e=>

[16-9481 GA Email Signature_social media-v2-01]<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_GeoscienceAus&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=9NqOivF_T3D--sYfCUlpX_rdt03KgTuuPZt0Cs4aVFc&e=>[16-9481 GA Email Signature_social media-v5]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_GeoscienceAustralia&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=3DRlLrkFxcqnXAuV715PbY894q4cRs3kjF92spHLDzo&e=>[16-9481 GA Email Signature_social media-v2-02]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_user_GeoscienceAustralia&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=7mJR46D3w_R6BAyDGhhcIY3Ou8rwHfzAWOHyMcZmPLM&e=>[16-9481 GA Email Signature_social media-03]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_geoscience-2Daustralia&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=mgQFXP6VbweL7Glrmsm8wYjFEmBsj7eyorBfFz7jbhA&e=>

Make flexibility work. If you receive an email from me at a time that is outside of normal business hours, I am sending it at a time that suits me. I am not expecting you to read or reply until normal business hours.

From: IGS-IC <igs-ic-bounces at lists.igs.org> On Behalf Of Nacho Romero
Sent: Wednesday, 13 November 2019 2:18 AM
To: ic IGS List <igs-ic at lists.igs.org>; igs-dcwg at lists.igs.org
Subject: [IGS-IC-846] Infrastructure Committee business

Hello to all of you,

In the end I will not make the trip to the AGU in SF this December and I cannot host the IC/DC meeting that I had asked you about a few days ago. On the other hand if there is quorum we could still hold the meeting if enough people are making the trip and the others we can join by telecon. Please let me know and I can get the IGS CB to find a meeting room and setup the webex.

Secondly I would ask you to please read the enclosed document from the CDDIS on the high-rate data long term archiving. This corresponds to Recommendation 2018-02 from last year's IGS Workshop. CDDIS will take the lead but we should have discussion and agreement that this appears OK to everyone. This way forward shall not be an imposition for the Data Centers (as far as I understand) but rather an alternative for storing the 96 high-rate station files each day in a much more convenient form, so this alternative shall be informed to the users and documented and explained on our different websites.

We have a volunteer to start to tackle recommendation 2018-04 and I am hoping that we have a plan to announce the switch away from the "compress .Z" to "gzip .gz" for IGS data and products, there are some issues to work through so if you have ideas or potential pitfalls please let us know.

This is related to the removal of the need to submit Rinex 2 files , for now we have kindly asked all station operators in the IGs to retain Rinex 2/ Rinex 3 capabilities for their stations and many have done so. some have already removed the Rine x2 data flows as they updated their data handling software and could not retain the old capabilities. By making a formal announcement soon that from 01/01/2020 IGS stations do not need to submit Rinex 2 any longer if the Rinex 3 station files are flowing , but that they can retain dual file submissions if they want, but then I would like to know if we can request that the Rinex 2 files be "gzip .gz" compressed so that we can move forward  ... please send your views on this. Thanks.

Please also look over the enclosed recommendations from the last two workshops and let us know if you would like to advance any one of them. Thanks. Otherwise I will keep trying to push for actions as needed.

Finally, the next IGS Workshop will be at UCAR in Boulder, CO on 14-18 August, 2020, I hope to see you there!  If there is no objection now (please email me) we will retain the current Infrastructure Committee composition and organization and bring it up for discussion next summer. I am open to any suggestions for changes including my Chairmanship, I have been at this for a while and enjoy it and would like to continue but if you would like to be considered we can have an internal IC process to elect a new Chair ahead of the next workshop.

Hope all is well at your end,

--

Best regards,

Nacho

_______________________________________________________________

Ignacio (Nacho) Romero

Aerospace Engineer, PhD

CSC Ltd @ ESA/ESOC/OPS-GN

Satellite Navigation Engineer @ ESOC Navigation Support Office

IGS Infrastructure Committee Chair

www.navigation-office.esa.int<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.navigation-2Doffice.esa.int&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=9gJ03KZm1uwACuREG3cMSxt3swgmBuf2bRuJpOZXZ6s&e=>

www.canaryspaceconsulting.co.uk<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.canaryspaceconsulting.co.uk&d=DwMGaQ&c=ApwzowJNAKKw3xye91w7BE1XMRKi2LN9kiMk5Csz9Zk&r=yrgx-xAXo5tjY0kqY8hF4m6wOxj3jNazq_pilwD5MO4&m=l3SpR0nHOeXROADi4Y-ftNmUnYuaDGJn1jxkPuCMNzM&s=DvbZxS1ni9zt2Cza2yqTdiCOAM46tZprg_F2N43wpxE&e=>

_______________________________________________________________

This message including any attachments may contain confidential

information, according to our Information Security Management System,

 and intended solely for a specific individual to whom they are addressed.

 Any unauthorised copy, disclosure or distribution of this message

 is strictly forbidden. If you have received this transmission in error,

 please notify the sender immediately and delete it.

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
-------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 24608 bytes
Desc: image001.png
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1237 bytes
Desc: image002.png
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1077 bytes
Desc: image003.png
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1914 bytes
Desc: image004.png
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 1228 bytes
Desc: image005.png
URL: <http://lists.igs.org/pipermail/igs-dcwg/attachments/20191118/c7ec5231/attachment-0009.png>