[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Duplicate obs (fwd)

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: Duplicate obs (fwd)
  • Date: Fri, 28 Apr 2000 12:33:42 -0600 (MDT)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Fri, 28 Apr 2000 14:13:52 -0400 (EDT)
From: C. Vandersip <address@hidden>
To: Arthur A. Person <address@hidden>
Subject: Re: Duplicate obs

While I don't know the cause, we have the same annoyances.  Fortunately,
in the "weather" program one can file the metar data in GnuDatabase (gdbm)
format using DBFILE and pqsurf in ldm.  This will eliminate duplicates
from showing up in the /OBS, /METAR, /METARDECODE, /METARFULLDECODE
commands in "weather".  To set this up:

1. Download and install gdbm-1.7.3 on your ldm server.  Make sure it's
this version, as the "weather" program is designed to work with it, and
newer versions (e.g. gdbm-1.8.1) will cause repeated failures in the data
processing under pqsurf. See my messages to ldm-support for a history of
our troubles using gdbm-1.8.1 and final solution (downgrading to 1.7.3).
In chronological order:

http://www.unidata.ucar.edu/glimpse/ldm/3558
http://www.unidata.ucar.edu/glimpse/ldm/3561
http://www.unidata.ucar.edu/glimpse/ldm/3563
http://www.unidata.ucar.edu/glimpse/ldm/3564
http://www.unidata.ucar.edu/glimpse/ldm/3565

2. Recompile ldm (we are using v5.0.8) to enable gdbm.  See

http://www.unidata.ucar.edu/glimpse/ldm/3185 for how to do this.

3.  Read section III-B of WEATHER's README file to configure LDM to store
the METARs in separate GDBM files.  Section III-C tells you how to set it
up to file both in ascii and gdbm formats (which we do here).

For reference, here's how our ldmd.conf and pqsurf.conf are set up for
gdbm:

--ldmd.conf--

#####
# $Id: ldmd.conf,v 1.7 1995/11/08 15:39:54 mitch Exp $
# Sample ldmd.conf for ldm5
####

#
# Programs that share a queue with rpc.ldmd
# are started by it and are in the same process group.
#

exec    "pqexpire"
exec    "pqbinstats -d /usr/local/ldm/logs"
exec    "pqact"
exec    "pqsurf -l /usr/local/ldm/logs/pqsurf.log -p ^S[AP] -Q  
/usr/local/ldm/data/pqsurf.pq /usr/local/ldm    /etc/pqsurf.conf"
<snip>

--pqsurf.conf--

WMO     ^metar (....) ([0-3][0-9])([0-2][0-9])
        DBFILE  /export/data/weather/METAR_GDB/(\2:yyyy)(\2:mm)\2\3.METAR_GDB \1
WMO     ^speci (....) ([0-3][0-9])([0-2][0-9])
        DBFILE  /export/data/weather/METAR_GDB/(\2:yyyy)(\2:mm)\2\3.METAR_GDB \1


Regards,

Chris



On Thu, 27 Apr 2000, Arthur A. Person wrote:

> Hi...
> 
> One of our users has brought to my attention that some (a lot?) of
> observations are sent as many as three times in raw WMO format on the IDD
> feed.  Perhaps not a big deal, but it's annoying when fed through programs
> such as "weather" to see the same ob show up three times.  Also, some are
> incomplete.  E.g.  KBOS for 000425@0054Z comes in under SAUS31, SAUS41 and
> SAUS80, and the SAUS31 ob does not contain the remarks section.  On the
> other hand some obs come in only once under entirely different headers,
> e.g. KUNV under SAUS70.  Anyone know the rationale behind this and what an
> appropriate selection method might be to get only one observation for a
> station without potentially throwing out other stations that would appear 
> under different headers?
> 
>                                          Thanks.
> 
>                                            Art.
> Arthur A. Person
> Research Assistant, System Administrator
> Penn State Department of Meteorology
> email:  address@hidden, phone:  814-863-1563
> 
> 

         ###############################################################
         #                      Chris Vandersip                        #
         #        Computer Research Specialist/Dept. Sysadmin          #
         #  Rm. 024, Dept. of Meteorology, Florida State University    #
         #          address@hidden   (850)644-2522                     #
         ###############################################################