[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #TTR-495389]: Slow access to local datasets through ADDE

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [IDD #TTR-495389]: Slow access to local datasets through ADDE
  • Date: Fri, 15 Feb 2008 14:20:06 -0700
Hi Samuel,

re:
> O.K. Just to let you know:
> 
> I have tried to reduce some of our data processing, to reduce how far behind 
> shu's
> processing times are, to eliminate gaps in the data we currently store.  
> Unfortunately
> it seems as though there are still gaps in the GINIEAST/GE1KVIS (GOES-EAST 
> 1km VISUAL)
> satellite imagery.

OK.  It may still be necessary to do some more tweaking of your processing 
(pqact) setup.

> Overnight, shu hasn't had any significant "Behind" messages in ldmd.log(only 
> 1/2 second or so),
> and pqmon -l- indicates the oldest queue entry is 2670 seconds.

This sounds much better.

> I have eliminated the ingesting of PR,AK, and HI imagery, and moved the 
> NNEXRAD file processing
> to a separate file system.  Unfortunately, this hasn't seemed to eliminate 
> all of our missing data.

You may want to create a pqact.conf_nnexrad file that only contains the NNEXRAD 
processing.
Then, remove NNEXRAD as one of the feedtypes in pqact.conf_mcidasB processing 
and add a
new ldmd.conf 'exec' line to run a new 'pqact' instance for the NNEXRAD 
processing:

change:

exec    "pqact -f FSL2|NLDN|NNEXRAD|UNIWISC|HDS|CONDUIT etc/pqact.conf_mcidasB"

to:

exec    "pqact -f FSL2|NLDN|UNIWISC|HDS|CONDUIT etc/pqact.conf_mcidasB"
exec    "pqact -f NNEXRAD etc/pqact.conf_nnexrad"

> Is there a way to prune down our data, to avoid processing data from outside 
> the lower 48
> United States?

This may be possible in theory, but in practice it would be pretty much of a 
nightmare.

As a first step, let's try moving the NNEXRAD processing into its own pqact 
invocation
as I outlined above.

If this doesn't do all that is needed, let's move the NIMAGE processing out of 
pqact.conf_mcidasA
into its own pqact.conf_nimage file and add another invocation of pqact to 
ldmd.conf
that just processes the NIMAGE data:

change:

exec    "pqact -f IDS|DDPLUS|HRS|NGRID|FNEXRAD|NIMAGE etc/pqact.conf_mcidasA"

to:

exec    "pqact -f IDS|DDPLUS|HRS|NGRID|FNEXRAD etc/pqact.conf_mcidasA"
exec    "pqact -f NIMAGE etc/pqact.conf_nimage"

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed