19991214: NLDN Y2K IDD header discussion (revived) (cont.)

>From: Unidata User Support <address@hidden>
>Organization: Unidata
>Keywords: NLDN IDD pqact.conf WMO header

David,

>     I share your concern about y2k issues and the NLDN feed.

>I've cc'ed this to Vince, Dave, and Pete in case they have some bright
>ideas on this.

OK.

>In addition to the exchange of email in May that you partially
>summarized, there was some further action on this in September. If
>you'll remember in May I emailed you:
>
>     I have good news and bad news. The good news is I think
>     I have sorted out what the various pieces of code do,
>     and located the source. The bad news it I can't get
>     any of it to compile!!!
>     Needless to say this leaves me more than a little uncomfortable!

Sandy has been able to get the package to build.

>     We have set up an account on our machines for testing
>     of the nldn stuff. The username in nldntest nldn41
>     on aspen.atmos.albany.edu. You mentioned you'd like to
>     look at the code to check for y2k compliance. Perhaps
>     you'd also be willing to see if you can get it to compile
>     at all. I just don't know enough c or c++ to get anywhere
>     with it myself. Any help you can give in this area would
>     be *most* appreciated.

Right.  Sandy has used this account.

>I can send you the bulk of that email that describes the various
>processes again if you'd like.

This would be helpful for Sandy.

>In September I talked with Dave Fulker about this at the PolComm
>meeting. He said he would ask around and see if anybody at UNIDATA
>wanted to look around at the code.  It was my hope that UNIDATA would
>agree that this data feed is of wide spread value to the community and
>would be able to help check it for us.

At that time I talked with Sandy Whitesel about this project and he
was and still is interested in digging in.

>Shortly thereafter somebody from Unidata was on aspen, and you wrote:
>
>     It was Unidata alright.  Sandy Whitesel (a part time system administrator
> )
>     has agreed to dig into the NLDN setup at SUNYA and Y2Kize things.  Sandy
>     has a lot more C++ experience than me, so he was a "warmer" body to
>     throw at the updating task.

Right.

>Unfortunately nothing seems to have come of this.

I wouldn't say nothing.  Sandy has been successful in getting the routines
to build, but nothing past this.  He will be working on the problem in
the next couple of days.

>As you know, we no longer have local expertise with this code or with
>C++. Also since this an unsupported effort no funds were available to
>hire a short term programmer to sort this our for us.

No problem.  We will be working on this.

>I've had to take what I call the Australian approach to y2k
>preparedness - that is, if it is mission critical or safety related
>then check it out and fix it. If it is not mission critical then wait
>to see if it breaks (since either way time will have to be spent to fix
>it, but this way no time was spent trying to check it before hand).

:-)

>Since the NLDN IDD feed is not mission critical (we don't make much use
>of it locally since we have other ways of obtaining and displaying the
>data) I've been forced to basically let it sit.

I would have done the same thing.

>Since lightning data is
>of less interest in January I hoped that this would allow us some time
>to scramble if needed and not have the community suffer too much.

This is exactly what Don said: there is basically no lightning in
January, so the pressure is not really on.

>I'm
>not happy about this - I'd rather know what was going to happen
>1/1/2000, and know that it will work.

Me too.  That is why Sandy will be working on the problem this week.

>Awhile back I was thinking that
>perhaps I'd send email to the users of this data stream and warn them
>that it might disapear on Jan 1

I would only do this if our efforts are not successful.

>(and maybe beg for contributions to
>keep it going), but it is really too late for that now.

>I see basically the following options at this point:
>1) we could send the email that you suggest
>2) we could wait to see what happens then deal with it
>3) we could check and perhaps fix the code then email
>   the community if changes are required.
>4) some combination of the above
>5) something else I haven't thought about

Us working on and fixing the code if needed?

>The problem with option 1) is that we do not really know how things
>will change on 1/1/2000.

We should later this week.

>I like to think that the header will change to 00JJJHHMbMe
>so sites would have to change their patterns to
> NLDN    ^([0-9][0-9]) ...
>As you say, it is also possible that the header will
>change to 100JJJHHMbMe, and sites will have to change their
>headers to something like
>NLDN    ^1([0-9][0-9]) ...
>Or the whole process could just break entirely.

Right, but headers in NOAAPORT change all the time, so users have
have to deal with that.  Since the lightning data is not mission critical
in January, I don't see this as being much of a problem.

>As best as I can tell, there are basically two processes 
>involved in sending this data to the ldm. 
>An ingest process that reads the data from the serial
>feed from satelite dish and wites it to a "raw file". This
>file is created with the 4digit year (hopefully determined
>correctly).

Excellent.  This process should continue to work into the new year.

>Then a nldnldm process reads from the raw file and creates 6 minute bin
>files. The file name for the bined data is generated by:
>strftime(n->binFileName, 8, "%y%j%H", begTm)

OK, the %y format option for strftime is the year within the century
[00,99].  This says that the file names _will_ go to 00JJJHHMbMe on
January 1.

>My understanding is that
>this should be OK on 1/1/2000 and will return a filename that starts
>0000100.

Right.

>The binned data is inserted in the LDM queue.  The product
>header is obtained in part from the bined file name.

OK.

>Of course any, of this could be completely wrong.

Just when you had me feeling warm and fuzzy, you drop the bucket of
ice water :-)

>So here is my best guess, and what I hope, will happen Jan 1 when we
>bring striker back up (all our machines are being shut down Dec 31
>since the University is shutting down networking gear there is little
>point in staying up, and, this provides additional protection should
>something really strange happen jan 1). The nldn feed will start back
>up, data will go out with yyjjjhh foremat where yy will be 00.

Given that the file name is created with the strftime function you list
above, this is what I believe will happen.  Sandy's work should verify
(refute) this AND give us a system that can be build/modified if/when
needed.

>Unless somebody with more experience and time wants to look at the code
>we are basically stuck with what we have for now.

Again, this will happen this week.

>I think a reasonable measure to take at this point might be to send
>email to the community saying two things:  That if they are using the
>pattern (9[3-9]) for the year that this will definitely fail on 1/1/00
>and they should change it to ([0-9][0-9]).

I don't disagree, but I would rather have people change their patterns
to something that would allow CCYY or YY.  This is, in fact, the
pattern I send out with the ldm-mcidas decoders.  I like either of
these names better than a file that is named YYYJJJHHMbMe.

>That the NLDN feed to UNIDATA affiliated Universities is a voluntary,
>unfunded, effort at UAlbany. That no resources are available to
>properly check the code for y2k readiness.  That we expect the feed
>will keep running (except for the days shortly before and after the
>switch when no UAlbany machines will be reachable from the outside
>world) and that the above pattern will match the feed header. But there
>are no guarantees the data will be in that format, or, that it will be
>available at all.

It seems reasonable to include information like this in a general
announcement to the community that also advises users how to alter
their LDM patterns.

>I'm open to other suggestions as well.

A couple fo  things:

o we can make the annuncement; this gets the onus off of you
o we will be looking at the code to make sure that it will keep
  working

How does that sound?

My note of yesterday was an attempt to not step on your toes in regards
to how the products are named.  It was not meant to raise red flags or
an indication that we are not going to work on this (several of your
comments make it sound like you do not belive this).

Tom