[IDD #QVQ-842701]: LDM of Seoul National University has problem

Hi Euihyun,

I apologize for taking so long to respond to your email.  A combination
of a small problem with our inquiry tracking system and my being away
at the AMS Annual meeting this week prevented a response before now.

re:
> I appreciated that your effort to set our LDM system up.
> My professor is Gyu-Ho Lim, who visited NCAR on last December.

No worries.

> However, our LDM system has some problems.
> We can find no data from upstream.
> And We find some error message from log file in data folder.
> I attached log file of my LDM system.

I took a look at your log file to see what the problem could
be.  In order to more fully investigate the problem, I decided to
logon to your machine, ssmgwdss.snu.ac.kr as user 'ldm'.

I see that you are receiving COSMIC data on the EXP feed in spite
of the errors you reported in from ~ldm/logs/ldmd.log file:

<as 'ldm'>
cd ~ldm
notifyme -vl- -f EXP -o 10000
Jan 16 21:00:36 notifyme[15777] NOTE: Starting Up: localhost: 
20090116181356.200 TS_ENDT {{EXP,  ".*"}}
Jan 16 21:00:36 notifyme[15777] NOTE: LDM-5 desired product-class: 
20090116181356.200 TS_ENDT {{EXP,  ".*"}}
Jan 16 21:00:36 notifyme[15777] INFO: Resolving localhost to 127.0.0.1 took 
0.00049 seconds
Jan 16 21:00:36 notifyme[15777] NOTE: NOTIFYME(localhost): OK
Jan 16 21:00:39 notifyme[15777] INFO:    15044 20090116195004.988     EXP 000  
/pub_out/cosmicrt/level2/wetPrf/2009.016/wetPrf_C005.2009.016.16.10.G11_0001.0002_nc
Jan 16 21:00:39 notifyme[15777] INFO:    15044 20090116195005.032     EXP 000  
/pub_out/cosmicrt/level2/wetPrf/2009.016/wetPrf_C005.2009.016.16.12.G23_0001.0002_nc
Jan 16 21:00:39 notifyme[15777] INFO:    15044 20090116195005.038     EXP 000  
/pub_out/cosmicrt/level2/wetPrf/2009.016/wetPrf_C005.2009.016.16.13.G15_0001.0002_nc
Jan 16 21:00:39 notifyme[15777] INFO:    15044 20090116195005.067     EXP 000  
/pub_out/cosmicrt/level2/wetPrf/2009.016/wetPrf_C005.2009.016.16.15.G10_0001.0002_nc
 ...

The problem is that there is no action defined for what to do with the
products once they are received.

A little bit of background is needed.  The LDM performs two primary
functions:

- move products from one machine to another

- do something with the products once they are received

The configuration needed to get products from an "upstream" (feeder) machine
is to add an appropriate line in the ~ldm/etc/ldmd.conf configuration file.
This was done on ssmgwdss.snu.ac.kr for two different streams of data:

- NOAAPort-delivered global observational data and low to moderate resolution
  model output (this was done mainly for testing purposes)

- GPS (radio occultation) data from COSMIC

The ~ldm/etc/ldmd.conf entries that make the appropriate requests are:

REQUEST WMO ".*" idd.unidata.ucar.edu
REQUEST EXP ".*" cosmic-io.cosmic.ucar.edu

The first entry requests the NOAAPort-delivered global observational data and
low to moderate resolution model output.  The second entry requests all of
the data provided by COSMIC.

I did not setup any processing actions for either of these types of data
when I installed the LDM when Dr. Gyu-Ho Lim was in Boulder because I did not
know what he/you wanted to do with the data.

Question:

- what would you like to do with the COSMIC GPS data once it is received
  at SNU?

In order to demonstrate that the data are being received, I just added
a simple FILE action to the "standard" pattern-action file, ~ldm/etc/pqact.conf.
The action I added is at the very end of the file and looks like:

#
# Example action added to write the COSMIC GPS data products to disk files
#
# Typical Product Headers:
#
# 
/pub_out/cosmicrt/level2/wetPrf/2009.016/wetPrf_C005.2009.016.19.12.G15_0001.0002_nc
# 
/pub_out/cosmicrt/level2/bfrPrf/2009.016/bfrPrf_C005.2009.016.16.10.G11_0001.0006_bufr
#
# History: 20090116 - Added FILE action for EXP-delivered products
#                       (Tom Yoksas, Unidata User Support)
#
EXP     (...Prf)_(.*)_(bufr|nc)
        FILE    -close  data/COSMIC/\1/\2_\3

NB: certain whitespace in pattern-action file entries MUST be tabs, not spaces:

- the whitespace between EXP and (...Prf) is a tab
- the whitespace before FILE is a tab
- the whitespace between FILE and -close is a tab
- the whitespace between -close and data/COSMIC is a tab

This creates the directories:

~ldm/data/COSMIC/wetPrf
~ldm/data/COSMIC/bfrPrf

In the ~ldm/data/COSMIC/wetPrf directory will be files that look like:

wetPrf_C004.2009.016.19.30.G24_0001.0002_nc
wetPrf_C004.2009.016.19.31.G29_0001.0002_nc
wetPrf_C004.2009.016.19.32.G19_0001.0002_nc
wetPrf_C004.2009.016.19.37.G02_0001.0002_nc
wetPrf_C004.2009.016.19.52.G07_0001.0002_nc
wetPrf_C004.2009.016.19.52.G25_0001.0002_nc
wetPrf_C004.2009.016.20.07.G05_0001.0002_nc
 ...

These files are in netCDF format.

In the ~ldm/data/COSMIC/bfrPrf directory there will be fills that look like:

bfrPrf_C004.2009.016.18.25.G26_0001.0006_bufr
bfrPrf_C004.2009.016.18.35.G05_0001.0006_bufr
bfrPrf_C004.2009.016.18.39.G19_0001.0006_bufr
bfrPrf_C004.2009.016.18.41.G10_0001.0006_bufr
bfrPrf_C004.2009.016.18.42.G30_0001.0006_bufr
 ...

These files are in BUFR format.

Because new data files will be written as data are received, I also turned on
data scouring so your disk would not eventually fill.  I did this by
adding an entry to 'ldm's crontab file that runs the LDM 'scour' utility once
per day:

MAILTO=""
#
# 20090116 - Run the LDM 'scour' utility at 9:30 local time (currently 0:30 UTC)
#
30 9 * * * bin/ldmadmin scour


I configured 'scour' by editing ~ldm/etc/scour.conf.  The scour.conf 
configuration
lines I added are:

#
# COSMIC data - keep 2 days of data online
#
~ldm/data/COSMIC                2


The LDM pattern-action file entries can be configured to do almost anything
you can imagine.  For more information on this and other LDM topics, please
read through the LDM documenation at:

http://www.unidata.ucar.edu/software/ldm

> Thank you for your help.

No worries.

Please let us know if/when you have questions on the LDM or on processing of
products received through actions in LDM pattern-action files.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************