[CONDUIT #NQZ-836725]: NOMADS Ingest

Glenn,

Looking at your grid, at present, CONDUIT carries
GFS 1 deg, GFS 0.5 deg, RUC20km.

The ensemble data on CONDUIT is now the GRIB2 pgrb2a that NCEP will be 
providing to tigge,
rather than the grib1 pgrba/b you have.

At this point, NCEP isn't placing the 13km RUC grid (grid #130) yet though it 
is on our list.

The RTMA is on NOAAPORT currently, as are a subset of the NAM 12km grids. We 
would
like to add the full 12km grids to CONDUIT inplace of the 20km grids currently 
sent.

In the future, we want to look at other data sets such as SREF and CFS for 
CONDUIT
after completing the transition of TOC file names to the NCEP file names. As 
you mention,
we would like to provide flaxibility in data sets, so some items such as 
delivering
the GFS in both the 1 degree and 0.5 degree for the same forecast hours and 
parameters
is using bandwidth that could be used otherwise. At the present time, we will
keep both to give users the GRIB1 option until they have fully transitioned to
GRIB2.

The primary point of your problem is that the LDM delivery is geared toward 
keeping
up to date- and if you fall behind due to network latency (or system 
performance),
you would drop data in order to catch up. That behavior works well with 
maintaining
realtime delivery, but does not match all the needs of a data archive center as 
well
since if you missed data, you would need an alternate mechanism for retireving 
the data.

The CONDUIT data is placed in the queue with a file inventory of the contents, 
so you would
be able to determine if you missed data and then you could use the oher system 
to
retrieve data should it be available, so it is possible to use CONDUIT to 
lessen the 
bottleneck, but given your mission to archive the data, the conflict with
LDM design means that you would still want another data route- but possibly 
that doesn't have 
to be the operational route. As an aside,information about what is on CONDUIT 
can be found here:
http://www.unidata.ucar.edu/data/conduit/ldm_idd/

As always, we are willing to look at how CONDUIT fits into the larger puzzle, 
including
NOMADS...

Steve Chiswell
Unidata User Support


> Hello Folks,
> The current ingest process for the NCDC NOMADS is achieved via a push
> from the NCEP mainframe backup computer (by Jordan) and has been listed
> as a potential operational bottle neck - and thus NCO has requested
> NOMADS to receive it's data elsewhere.  CONDUIT is an option.  At the
> last  C2 meeting I read Brent's NCEP Status report (Attached) found at:
> http://www.unidata.ucar.edu/data/conduit/c2summary_060131.html
> 
> but it's not clear to me what data sets actually on CONDUIT or not.
> I'm sure this list is somewhere....
> 
> At the 20K foot level- an Archive- (NOMADS) needs to consider the long
> term use and the requirements of the various datasets to store- not just
> from an operational real-time perspective and I find it a difficult
> decision to make- if I go with CONDUIT I will be at the mercy of the
> "community" regarding what's on the that particular feed.  While serving
> the community is the #1 goal- I thought it best to solicit opinions
> regarding how the NOMADS Archive will be populated.  Attached is a list
> of what we archive and plan to archive.  Also- notice that the NARR is
> not to be found that we can tell on these lists.  As the NOAA access
> point for the NARR- this of course is of concern.
> 
> Our NCEP Real-Time Mirror Server is also very popular (about 5Tb/month
> 5M downloads/mo), and it has a real-time _need_  but not a formal
> requirement.  First and foremost we are an archive.  While I would like
> to continue this real-time service- with the significant latency I see
> on the NOC, I'm not sure how best to archive this goal.  CONDUIT looks
> fast however in this regard.
> 
> Maybe multiple sources of ingest is the answer- although that's not
> something I want to maintain over time.  We do have NOAAPort here and
> that could supplement what's not on CONDUIT and that along with the ftp
> servers at NCEP and "NWS" we'll have to merge our requirement.
> 
> I would appreciate any and all comments regarding NOAA's and Unidata's
> user needs, thoughts on what to archive, Ingest options for high
> priority queuing at ftpprod or tg, and maybe a forum to solicit those
> user needs.
> 
> -Regards, Glenn
> 
> --
> Glenn K. Rutledge
> Services Team Leader
> Remote Sensing and Applications Division
> NOMADS Project Manager
> National Oceanic and Atmospheric Administration
> National Climatic Data Center
> Asheville NC 28801
> Phone: (828) 271-4097
> Fax: (828) 271-4328
> 
> NOMADS: http://nomads.ncdc.noaa.gov/
> 
> 
> 


Ticket Details
===================
Ticket ID: NQZ-836725
Department: Support CONDUIT
Priority: High
Status: Closed