Re: NCEP TIGGE data to ECMWF through NCAR

Doug,

I have been reviewing ultrazone's ldmd.conf file request lines related
to CONDUIT,
and have found a problem that can result in lost products as you are
seeing:

You have 2 sets of request lines to idd.unidata.ucar.edu and
idd-cise-nsf.gov
with the patterns:

REQUEST CONDUIT "data/nccf/com/gens" 
REQUEST CONDUIT "TIGGE" 
REQUEST CONDUIT "^.status\.*data/nccf/com/gens"
REQUEST CONDUIT "^.status.*TIGGE" 

The problem here is that the first 2 lines are supersets of the 3rd and
4th lines.
That is, the unanchored substring (line 1) "data/nccf/com/gens" in the
first line will
also appear in the .status products matched by the 3rd line, and 
the unanchored substring "TIGGE" (line 2) will also appear in
the .status products matched by the 4th line. The complication is that
these are also coming from 2 unique
hosts, which is part of the "don't do this....or this..." examples in
the LDM tutorial:
http://www.unidata.ucar.edu/software/ldm/ldm-6.4.6/basics/configuring.html#ldmd.conf

Because your host is disconnecting occasionally when switching back and
forth between primary and alternate hosts, the reconnection will use the
most recent
product in the queue that matches the request line. The relatively
small .status products will arrive faster in the anchored request line,
than the unanchored superset line, so you have the possibility that the
reconnection will start from the time of the .status  message and skip
data products that are requested. Note that
your ldmd.log files show that most of the primary/alternate host
switching is occuring
with the small status files, and not the larger superset request lines-
so your
missing data products are clumped in small time preriods rather than
continuously.

You should ensure that your request lines are completely disjoint by
anchoring the the first 2 request lines to eliminate the  overlap with
the status lines, eg
REQUEST CONDUIT "^data/nccf/com/gens" 
REQUEST CONDUIT "^data2/TIGGE" 

Or, probably better to just eliminate the 3rd and 4th lines above in both sets 
of upstream requests
since fewer writers locking the product queue regions is better.


Steve Chiswell
Unidata User Support





On Tue, 2007-02-27 at 20:15 -0700, Doug Schuster wrote:
> On Tue, 27 Feb 2007 12:45:54 -0500
>   Brent A Gordon <address@hidden> wrote:
> > Hi Doug,
> > 
> > I'm very sorry this is coming so late.  I was out of the 
> >office when you sent this and i have yet to make it back 
> >through all of my email that came in at that time.
> > 
> > I believe there were one of two solutions to this issue. 
> > One where NCAR does the product reformatting as you 
> >specified below.  The other where ECMWF duplicates this 
> >processing.  NCEP does not have a preference for which 
> >method is employed.  As long as NCAR and ECMWF agree on a 
> >method, NCEP should be good.
> > 
> > That is the correct email address to send retransmission 
> >requests to.
> > 
> > Brent
> > 
> Brent,
> 
> If NCAR re-activates it's retransmission request scripts,
> will the requests now be processed by NCEP?
> We are using the most current version of ldm (6.5.1), and 
> Unidata
> staff have reviewed NCAR's ldm configuration as requested 
> by NCEP staff
> last week.
> 
> See Below:
> 
> From:         Joe Carr <address@hidden>
> Subject:      Re: [NCEP.List.PMB-PCSP] TIGGE resend requests.
> Date:         Thu, 22 Feb 2007 09:53:10 -0500
> To:   Doug Schuster <address@hidden>
> Cc:   Patrick O'Reilly <address@hidden>, Paula 
> Freeman <address@hidden>, PMB Dataflow Team 
> <address@hidden>, Steve Chiswell 
> <address@hidden>
>       
> 
> Doug,
> 
> The resend of these data will have an impact on other 
> users. We have not had any problems reported with any 
> other customers receiving data via the ldm. At this point, 
> can we ask that a bit more intensive investigation go 
> forward into why your ldm is not receiving and/or 
> processing the full complement of data rather than NCEP 
> resending data which could/will impact other customers. I 
> believe that Steve Chiswell's email was a good place to 
> start.
> 
> Thanks,
> Joe Carr
> 
> Doug Schuster wrote:
> 
>      Hi Patrick,
> 
>      Is NCEP actively resending the missing fields we've 
> been requesting since last week?
>      I haven't noticed any of the requested data come in 
> starting with 2007021518.
>      FYI, I switched the software so it will only send one 
> email request per resend without continued requests
>      if the data has not yet arrived.
> 
>      Doug
> 
> 
> > Doug Schuster wrote:
> >> Hi Brent,
> >>
> >> Baudouin Raoult at ECMWF needs you to verify that NCEP 
> >>has agreed to 
> >> an arrangement where NCAR captures
> >> NCEP's ensemble data off CONDUIT, patches the GRIB-2 
> >>headers to comply 
> >> with TIGGE standards,
> >> and, when/if resources become available, recomputes 
> >>accumulation 
> >> parameters to represent values from start of forecast. 
> >>
> >> Also, can you verify that missing fields will be resent 
> >>via CONDUIT to 
> >> NCAR according to requests sent to 
> >> address@hidden 
> >><mailto:address@hidden>.
> >>
> >> After you verify these items, ECMWF will discontinue 
> >>extracting 
> >> partial data directly from CONDUIT, and switch solely to 
> >>the NCAR
> >> feed for processed, TIGGE compliant, NCEP data.  And at 
> >>this point, 
> >> NCEP data will be officially included in ECMWF's TIGGE 
> >>archive.
> >>
> >> Thanks -Doug
-- 
Steve Chiswell <address@hidden>
Unidata