This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Ben, re: > We're trying to do two things with the AWOS surface data that comes in > off of the LDM. First, we want to send the data to a processing script > which will store the data in an SQL database, and second we want to also > store the information in its raw form in separate files labeled by > station id. Below are an excerpt from our pqact.conf file in which we > attempt this. I thought this was working (its been this way since > July), but we recently discovered that there are several files missing > that we expected to be saving as part of the second action we're taking > on the data (saving in the raw form). For example, we aren't seeing > KLGU.metars, and the data for KLGU isn't in any of the 116 > files/stations that are there. It sounds like you did not receive data for KLGU. > Can you offer any help or advice? I see nothing wrong with your second pqact.conf actions, so I don't have much, if anything, to offer there. > I was thinking perhaps the ldm > didn't like having two exact patterns to match (ie, the first pqact > entry was "stealing" stuff from the second). Your pqact.conf can have as many actions that match a product as you like. The processing of the patterns proceeds from top to bottom through the actions in the pqact. The only way that a second (third, fourth, etc.) action would not be processed is if the product got deleted from the queue before the product was processed or if the second (or third, fourth, etc) action were incorrectly formated. It is not very likely that the product in question would get deleted between the end of one processing action and another that immediately follows it in pqact.conf (but it is possible, I guess). If you are convinced that your first pqact.conf action is processing a product that is not being processed by a second action, you might review how large your LDM product queue is to make sure that it is large enough to allow time for processing products before they get removed by the arrival of new products. One good way to determine if your product queue is too small is to use the LDM 'pqmon' utility. The last piece of information that 'pqmon' prints is the age of the oldest product in the LDM queue. If this age is _very_ small (like a few seconds), then it is likely that products are being scoured out of your queue before they are processed by all the pqact.conf actions you have. Also, if you have LOTS of actions in your pqact.conf file (hundreds or thousands) you may want to run more than one invocation of 'pqact' out of your ~ldm/etc/ldmd.conf file specifying different pqact.conf pattern/action files (GEMPAK users do this routinely). The key to using more than one pqact.conf file (will need to be named differently) is to remove processing actions out of one and put it/them in the other so they don't get executed more than once. Questions: - how large is your LDM product queue - how much data are you ingesting - how many 'pqact' processes do you have running trying to process your data - how many actions do you have in your pqact.conf file(s) - what is the age of the oldest product in your LDM queue > Thanks! No worries. > <portion of the pqact.conf:> > > # Metars > > # Append all US metars. > > # This action will slowly consume disk space. > > IDS|DDPLUS ^SAUS.. .... (..)(..) > > PIPE -close -strip > /usr/local/ldm/bin/processMETARnow.pl > > > > # Archive all records to a single file per station for backup purposes > > IDS|DDPLUS ^SAUS.. (....) > > FILE -close data/awos/\1.metar Assuming that your pqact.conf actions are properly formatted (e.g., use of tabs instead of spaces where needed), I see nothing wrong/alarming. If your processing problem is related to your processMETARnow.pl Perl script, there is not much we can comment on since we don't know what your script does. Cheers, Tom **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: YCI-227804 Department: Support IDD Priority: Normal Status: Closed