[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[AWIPS #KEZ-434248]: /awips2/data_store/manual directory

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [AWIPS #KEZ-434248]: /awips2/data_store/manual directory
  • Date: Fri, 16 Sep 2022 12:32:12 -0600
Hey Pete,

> Greetings. As we ingest grib2 data (by pulling a grib2 file from a NOAA Server
> and placing a copy in /awips2/data_store/ingest/) the data is processed 
> normally.
>
> But we see a directory /awips2/data_store/manual/grib/YYYYMMDD/HH/ (where 
> YYYYMMDD
> is today's date and HH is the present ZHR) with copies of the grib files we
> originally placed in /awips2/data_store/ingest/.
>
> The files in /awips2/data_store/manual/grib/ are taking up a sizable amount of
> space and after manually deleting these they did not impact how data is 
> rendered
> in CAVE (because we didn't touch the hdf5 files).
>
> Assuming the files in the "manual" dir are obsolete once the hdf5 data is 
> created,
> is there a config file that will direct EDEX to immediately remove the data 
> in the
> "manual" dir once the hdf5 file is created?

The manual directory is specifically intended to use in a "test" format, and 
should 
not be used for "production" or continual ingest of datasets.  The main reason, 
is 
exactly like you've highlighted, the manual endpoint copies and uses 
significantly 
more resources than you'd want to use for a production dataset.
Instead of placing the files in the manual endpoint, can you change your process
(I'm assuming you have some kind of script running) to use the 
notifyAWIPS2-unidata.py
script to insert the data into EDEX instead?  This python script is located in:
  /awips2/ldm/dev/notifyAWIPS2-unidata.py
Let me know if that makes sense, and if that's a possibility for you?
Thanks!

--Shay Carter

She/Her/Hers
AWIPS Software Engineer
UCAR - Unidata

If you're interested, please feel free to fill out a survey about the support 
you receive: 
https://docs.google.com/forms/d/e/1FAIpQLSeDIkdk8qUMgq8ZdM4jhP-ubJPUOr-mJMQgxInwoAWoV5QcOw/viewform

Ticket Details
===================
Ticket ID: KEZ-434248
Department: Support AWIPS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.