[McIDAS #JIN-832252]: cannot get MySQL database working with GRIB data

Hi Robert,

First, I must apologize for not getting back to you on your other inquiries.
Just so you know, it is workshop time here in Unidata, and I am, as usual,
struggling to get the materials and machines ready for my session which starts
on Monday.

re:
> I started a new support thread because this about the new MySQL GRIB server
> specifically.

OK.

> I wanted to make sure I could use standard NOAAport data before ruling out
> being able to use AMPS data this way.

Good thinking.

> I did the following:
> 
> 1) Installed mysql4 from www.blastwave.org under Solaris 10 x86.  This is
> a binary install and installs in /opt/csw/mysql4.  I started the database
> using the svcadm method and also set passwords per the MySQL docs.  I verified
> with svcs -a that the database is running.

OK, so far so good.

> 2) Did a make clobber on my McIDAS install and went back and did:
> make all -mysql=/opt/csw/mysql4 VENDOR=-vendor
> then did a make install.all
> I had no errors.

Hmm... It might be the case that the -mysql flag value is not being used
correctly in the 'makefile'.  To verify if it is being used correctly,
please check your ~mcidas/mcidas2006/src/makelog file for the compilation
of routines used in GRIB processing.  This will include, but may not be
limited to, xcdgrib.c in the main McIDAS section, and all of the C routines
in the XCD section.  If the -mysql flag is not being used correctly,
I will have to figure out what modification I need to make to have things
work like they do in the SSEC makefile.

> 3) Edited gribadmin script so that it matches my setup (bloastwave uses
> Sun compilers to compile mysql, not gcc:
> 
> ###########################################################
> # This script is to be used for the administration of the #
> # MySQL database used in conjunction with the GRIB filer. #
> ###########################################################
> 
> #=========================================================
> # Configuration Section:
> #
> # The variables in this section may need to be modified
> # to conform to your system.
> 
> # Database and user names for MySQL calls
> database_name="mcrtgrib"
> database_user="gribread"
> 
> # Path to the MySQL executable
> mysql_path=/opt/csw/mysql4/bin
> 
> # Path to gcc shared libraries
> gcc_lib_path=/opt/SUNWspro/lib
> 
> #=========================================================
> 
> 
> # Add MySQL to the PATH
> PATH=$PATH:$mysql_path
> export PATH
> 
> # Add path to gcc shared libararies
> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$gcc_lib_path:/opt/csw/mysql4/lib
> export LD_LIBRARY_PATH
> 
> 
> 4) From command line ran:
> gribadmin makedb
> It asked me for the password I setup the mysql user to have, and it
> did create mrctgrib in my mysql db directory:
> 
> drwx------   5 mysql    mysql        512 Oct 26 13:45 .
> drwxr-xr-x   8 root     bin          512 Oct 25 17:27 ..
> -rw-rw----   1 mysql    mysql    5242880 Oct 25 17:28 ib_logfile0
> -rw-rw----   1 mysql    mysql    5242880 Oct 25 17:28 ib_logfile1
> -rw-rw----   1 mysql    mysql    10485760 Oct 25 17:28 ibdata1
> drwx------   2 mysql    mysql        512 Oct 26 13:45 mcrtgrib
> -rw-r--r--   1 mysql    mysql       2468 Jul 29 15:50 my.cnf
> drwx------   2 mysql    mysql       1536 Oct 25 17:27 mysql
> -rw-rw----   1 mysql    mysql          6 Oct 25 17:28 mysql.pid
> -rw-rw----   1 mysql    root        1019 Oct 25 17:28 psnwx.err
> drwx------   2 mysql    mysql        512 Oct 25 17:27 test
> # pwd
> /opt/csw/mysql4/var

OK, but the question remains if the McIDAS routines were actually comipiled
with the mysql infomation.

> 5) used DECINFO to set al inactive, except for DMBIN whihc is set to active.
> 
> 6) Copied xcd_run to ~ldm/decoders and modified as needed.
> 
> 7) Edited ldmd.conf to ass EXEC xcd_run MONITOR
> 
> 8) Edited my pqact file as such:
> 
> EXP|HDS .*
> PIPE    xcd_run HRS
> 
> 9) Created directoty /data/ldm/mcidas/grib and made sure redirect was correct:
> *.gr* /data/ldm/mcidas/grib
> redirect.k: Done

These steps look correct.

> 9) I then stopped and restarted the LDM.
> 
> 10) dmbin.k is running, and there are files in /data/ldm/mcidas/grib that are 
> put there by McIDAS:
> 
> > ls
> AWC.190.2006299.1300.1.255.grib    NAM.84.2006299.1200.30.207.grib
> AWC.190.2006299.1348.0.255.grib    NAM.84.2006299.1200.30.211.grib
> AWC.190.2006299.1353.0.255.grib    NAM.84.2006299.1200.30.212.grib
> AWC.190.2006299.1358.0.255.grib    NAM.84.2006299.1200.30.215.grib
> AWC.190.2006299.1403.0.255.grib    NAM.84.2006299.1200.30.216.grib

OK.  This portion of the processing chain is separate from the actual
use of the GRIB files by the server.

> 11) I used DSSERVE to add entry for the NAM:
> 
> DSSERVE ADD NAM/GRID212 GRIB TYPE=GRID DIR='/data/ldm/mcidas/grib/NAM*212*'
> 
> 12) I ran GRDDISP and was able to display data from this set, however I 
> noticed access was slow, so I ran:
> gribadmin latest
> and this returned nothing.

This tells me that the mysql info was _not_ passed to the routines that need
it.  I still would like you to verify this by reviewing your makelog entries.

> 13) I checked XCD_START.LOG and indeed it said that MySQL was not being used:
> 
> Starting MONITOR at 06299.135123
> processing from spool file  HRS.SPL      RTMODELS.CFG
> DMBIN        Starting: GRIB/BUFR filer
> startxcd.k: m0monexe - first startup of :DMBIN                                
>     15034
> DMBIN:
> DMBIN: Invalid Processed pointer.
> DMBIN: first POINTER= argument is too small --> -2139062144
> DMBIN: Must be valid integer value within range 4096 thru 16781312.
> DMBIN:
> DMBIN: GRBFILER.PRO not found. Processed pointer set to last byte written.
> DMBIN: Not using MySQL database for GRIB
> decoder error    -3 message at      13096354  6299 135134
> decoder error    -8 message at       1941391  6299 135257
> decoder error    -8 message at      11449977  6299 135741
> decoder error    -8 message at        899321  6299 140259
> 
> Obviously GRDDISP is just reading the GRIB directly without using the 
> database.

Yes.

>This won't work for me as it's too slow.

OK.

>Any ideas where I went wrong?

The mysql stuff was most likely not passed in for the compilation step.

> Obviously GEMPAK is superior to McIDAS for model data, but there are
> some advantages to using McIDAS with model data and satellite imagery
> together so

OK.  I can see benefits in having both datasets available in the same
package.

> if the MySQL method is fast enough I'd like to give it a try.

OK.

> Reading GRIB messages directly is not fast enough.

I understand.  My problem is that I do not have much if any free time at
the moment to delve into what could be missing in the McIDAS makefile.

As I understand things, you are heading off for Antarctica "soon" so
your time is short.  What I don't know is what sort of timeframe we are
talking about.  My workshop starts on Monday and ends on Thursday
afternoon.  After that, I will should have some time to dive into the
makefile to see what is missing.  The problem with that is that I am
helping out with the LDM workshop on Friday and Saturday, and then
I fly to Washington, DC for an AGU meeting on Nov. 6-7.

Now, without going through all of the emails you sent in previously in detail,
can you let me know if you have attempted to decode the AMPS data using
the DMGRID method?  I recall from a quick skim of one of your messages
that one of the questions was where the AMPS model data would be decoded.
Since I can't answer that off of the top of my head, I have to offer the
following:

If you setup McIDAS-XCD grid decoding (i.e., DMGRID) to process _only_ the
AMPS data, then it should become obvious which file(s) the data is being
decoded into _if_ they are decoded at all.  Given the setup in RTMODELS.CFG,
I would expect that the AMPS data would be decoded into the SCRATCH set, or
GRID files 5001 - 5010, again _if_ they are decoded at all.  So, I would
setup a test system to:

- only pipe the AMPS data into 'xcd_run HRS' in pqact.conf
- make sure you are ingesting the AMPS data you want through an appropriate
  ldmd.conf request
- make sure that the test system is not already looking at a number of 
  GRID files into which the output could be put
- startup XCD
- monitory the log output in XCD_START.LOG
- see if the AMPS data gets decoded

A short comment here: if the DMGRID method does not work, then it means
that the GRIB serving method won't work either.  After all, it is the
same code used to decode the GRIB 1 messages and file them into GRID files
as it is to decode the data from GRIB 1 messages on the fly by the grib
server.

Now, if the AMPS data does get decoded correctly, then you should be able
to use GRDLIST with the FORM=ALL keyword to list out the originating center,
parameter ID, etc. so that you can setup an RTMODELS.CFG entry(ies) to file
the data into a GRID file(s) of choice.  The pieces that will be needed
to be addressed at this point are:

- REDIRECTion entries to point at your desired output GRID files
- DSSERVE additons to create a dataset(s) to access the decoded data

Of course, the downside of decoding using the DMGRID approach is the size
of the output GRID files.

Please let me know your thoughts at this point...

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JIN-832252
Department: Support McIDAS
Priority: Normal
Status: Open