[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Reingestion problem at LSU
- Subject: Re: Reingestion problem at LSU
- Date: Tue, 22 Jan 2002 11:44:21 -0700
Robert Leche wrote:
>
> Hello Unidata, Anne and all,
>
> I observed something related to a problem I reported yesterday. I
> found that data is being parsed out of the LDM queue as the pqact.conf
> tree defines 'what goes where'. But the dates on the files where
> re-ingested where wrong. The date number part (of the original raw
> file date) was correct, but the year and month parts where wrong,
> showing up as the current year and month. Mistral's pqact.conf is
> attached.
>
> Selected examples of files re-ingested into our digested data product
> tree for January 16 ( Raw files generated 09/26/2001):
> Model products:
> -rw-r--r-- 1 ldm5 data 11652236 Jan 16 13:17 02012600.mod
> -rw-r--r-- 1 ldm5 data 5476036 Jan 16 14:22 02012612.mod
> Extra products:
> -rw-r--r-- 1 ldm5 data 75064 Jan 16 14:21
> 020126.SRCC.summary
> -rw-r--r-- 1 ldm5 data 563567 Jan 16 21:08
> 020126.climate
> -rw-r--r-- 1 ldm5 data 43699336 Jan 16 21:08 020126.extra
> -rw-r--r-- 1 ldm5 data 2968501 Jan 16 21:07
> 020126.rainavg
> -rw-r--r-- 1 ldm5 data 55564356 Jan 16 21:08
> 020126.rainfall
> -rw-r--r-- 1 ldm5 data 8445 Jan 16 15:11
> 020126.rainrep
> -rw-r--r-- 1 ldm5 data 33081 Jan 16 15:07
> 020126.satellite
> -rw-r--r-- 1 ldm5 data 3558468 Jan 16 21:08 020126.sfcfor
>
> -rw-r--r-- 1 ldm5 data 2678798 Jan 16 21:08 020126.ship
> -rw-r--r-- 1 ldm5 data 1382776 Jan 16 21:08
> 020126.shiprpt
> -rw-r--r-- 1 ldm5 data 375025 Jan 16 21:08 020126.ships
> -rw-r--r-- 1 ldm5 data 242609 Jan 16 21:06
> 020126.summary
> -rw-r--r-- 1 ldm5 data 2122028 Jan 16 21:06
> 020126.tropical
> -rw-r--r-- 1 ldm5 data 589226 Jan 16 21:07
> 020126.tropics
> -rw-r--r-- 1 ldm5 data 3002 Jan 16 14:23 020126.upair
> -rw-r--r-- 1 ldm5 data 936 Jan 16 13:16
> 02012600.reconn
> -rw-r--r-- 1 ldm5 data 396 Jan 16 13:20
> 02012601.reconn
> -rw-r--r-- 1 ldm5 data 1846 Jan 16 14:21
> 02012612.reconn
> -rw-r--r-- 1 ldm5 data 496 Jan 16 14:24
> 02012613.reconn
> -rw-r--r-- 1 ldm5 data 468 Jan 16 21:07
> 02012618.reconn
> -rw-r--r-- 1 ldm5 data 240 Jan 15 18:27
> 02012620.reconn
> -rw-r--r-- 1 ldm5 data 488 Jan 15 19:16
> 02012622.reconn
> -rw-r--r-- 1 ldm5 data 158 Jan 15 19:30
> 02012623.reconn
>
> An example of a proper 'file date' format should be 010926xx.abcdef.
>
> I will away from the phone between 11:00am - 2:00pm (central time)
>
> Ideas?
>
> Bob
>
> -----------------------------------------------------------
>
> The following text where reported yesterday:
>
> -----------------------------------------------------------
> Hello Anne,
> It been awhile since problems have creep up and this is a good thing.
>
> All is well with real time data acquisition at LSU, but I am running
> into problems reingesting data on mistral.srcc.lsu.edu with pqing.
> For years, the following script was used to reingest raw data.
>
> #!/bin/csh
> # run this as root on mistral only!
>
> cd /sirocco/rawfiles
>
> # Clone this line and cut to hours needed...don't run too many at once!
> #foreach hr ( 00 01 02 03 04 05 06 07 08 09 10 11 )
> #foreach hr ( 12 13 14 15 16 17 18 19 20 21 22 23 )
>
> foreach day ( 26 )
> #foreach hr ( 13 14 15 16 17 18 19 20 21 22 23 )
> foreach hr ( 11 12 13 14 15 16 17 )
> # if(($day == 10) && ($hr <= 10) ) continue
> foreach fi (`ls 200109$day$hr.raw`)
> set FN=`echo $fi | cut -d. -f1,2`
> echo "Reingesting $FN"
> pqing -l - -f ddplus $FN
> echo "Sleeping..."
> sleep 90
> end
> end
> end
>
> When the script is invoked, the output looks normal:
> Sleeping...
> Reingesting 2001092612.raw
> Jan 16 20:15:34 pqing[757509]: Starting Up
> Jan 16 20:15:34 pqing[757509]: FILE "2001092612.raw"
> Jan 16 20:19:58 pqing[757509]: Lone ETX error
> 9 -01 00 162019
> Jan 16 20:21:06 pqing[757509]: Lone ETX error
> 9 -01 00 162021
> Jan 16 20:21:20 pqing[757509]: Lone ETX error
> 137 -01 FAGG26 1200 162021
> Jan 16 20:21:21 pqing[757509]: Lone SOH error
> 514 288 SAHO31 MHTG 261200 /pMETAR
> Jan 16 20:21:21 pqing[757509]: Lone ETX error
> 691 -01 FADN26 1200 162021
> Jan 16 20:21:29 pqing[757509]: Lone SOH error
> 258 888 SMGY01 SYCJ 261200
> Jan 16 20:22:15 pqing[757509]: Exiting
> Jan 16 20:22:15 pqing[757509]: Queue usage (bytes):214410488
> Jan 16 20:22:15 pqing[757509]: (nregions): 65505
> Jan 16 20:22:15 pqing[757509]: Duplicates rejected: 0
> Jan 16 20:22:15 pqing[757509]: WMO Messages seen: 10822
> Jan 16 20:22:15 pqing[757509]: SOH/ETX missing : 6
> Jan 16 20:22:15 pqing[757509]: parity/chksum err: 0
> Jan 16 20:22:15 pqing[757509]: WMO format errors: 0
> Jan 16 20:22:15 pqing[757509]: FILE Bytes read: 11050352
> Sleeping...
> Reingesting 2001092613.raw
> Jan 16 20:23:45 pqing[763240]: Starting Up
> Jan 16 20:23:45 pqing[763240]: FILE "2001092613.raw"
> Jan 16 20:24:04 pqing[763240]: Lone ETX error
> 263 -01 FADN26 1200 162024
> Jan 16 20:25:05 pqing[763240]: Lone SOH error
> 258 105 SAHO31 MHTG 261300 /pMETAR
> Jan 16 20:25:09 pqing[763240]: Exiting
> Jan 16 20:25:09 pqing[763240]: Queue usage (bytes):224751256
> Jan 16 20:25:09 pqing[763240]: (nregions): 72374
> Jan 16 20:25:09 pqing[763240]: Duplicates rejected: 0
> Jan 16 20:25:09 pqing[763240]: WMO Messages seen: 6019
> Jan 16 20:25:09 pqing[763240]: SOH/ETX missing : 2
> Jan 16 20:25:09 pqing[763240]: parity/chksum err: 0
> Jan 16 20:25:09 pqing[763240]: WMO format errors: 0
> Jan 16 20:25:09 pqing[763240]: FILE Bytes read: 4864625
> Sleeping...
>
> I've set the -v verbose flag and gathered the following messages:
> Produces a message that the product is queue.
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]: 110 20020116200050.071 DDPLUS 892
> SACN89 CWAO 260000 RRC
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]: 77 20020116200050.073 DDPLUS 893
> CSCN03 CWVR 260000
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]: 3480 20020116200050.076 DDPLUS 901
> USUS50 KWBC 260000 RRE
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]: 3324 20020116200050.081 DDPLUS 902
> USUS50 KWBC 260000 RRF
> Jan 16 20:00:50 pqing[672520]: Product already in queue
> Jan 16 20:00:50 pqing[672520]: 350 20020116200050.085 DDPLUS 903
> USUS50 KWBC 260000 CCB
> Jan 16 20:00:50 pqing[672520]: Product already in queue
>
> So, the ingested data is making it to the ldmqueue but looks as if these
> data are not processed out of it. The results of this is a file system
> tree that should be expanded with the reingested data but is not. This
> is strange, as I said before as the pqing process use to work. I already
> tried deleting and remaking the ldmqueue, no difference. Also, 'ldmadmin
> watch' has not revealed data processed from the selected date only
> current ingestion. We are processing real-time data with pqact. Mistral
> is an SGI IREX system running Ldm version ldm-5.0.8.
>
> --
>
> ----------------------------------------------------------------
> Robert Leche
> System Administrator
> Louisiana State University - Southern Regional Climate Center
> 260 Howe-Russell Building
> Baton Rouge, La. 70803
> address@hidden
> 225 578 5023
> ----------------------------------------------------------------
Hi Bob,
I apologize for not responding sooner. I was at the AMS conference in
Orlando last week.
Let me make sure I understand your problem correctly. In the example
you gave me you were trying to ingest a stream of products from a raw
file that were presumably created in September, but it appears that
pqact created files from the products in the raw file that had the
current date.
Are you sure this raw file actually contains products that were created
in September? It is possible that even though the filename indicates a
September date, such as '2001092612.raw', the products contained in the
file are from some other date. Your pqact.conf entries, such as
DDS|PPS ^WHXX.* .... ([0-3][0-9])
FILE data/ddplus/severe/(\1:yy)(\1:mm)\1.mod
and
DDS|PPS ^FO.* .... ([0-3][0-9])([0-2][0-9])
FILE data/ddplus/model/(\1:yy)(\1:mm)\1\2.mod
are creating file names based on the product bulletin time. So if
2001092612.raw contained products from the current day, that's how they
would be filed.
Or, are the September products possibly being filed somewhere where you
aren't finding them while new products are also being filed?
Anne
--
***************************************************
Anne Wilson UCAR Unidata Program
address@hidden P.O. Box 3000
Boulder, CO 80307
----------------------------------------------------
Unidata WWW server http://www.unidata.ucar.edu/
****************************************************