[McIDAS #HDN-934891]: ERAU ADDE configuration and Conference call to discuss equipment purchase

Hi Chris,

Tom here...

re:
> I am in the process of setting up a new installation of the McIDAS
> software to prepare for GOES-16 operational data.  This email has a
> couple of questions and a couple of requests related to this effort.

OK, that is why I drove this question into the McIDAS area of our
inquiry tracking system.

re:
> Fred Mosher helped us build some Goes-East and GOES-West composite images,
> with the use of multiple channels to create our "Vis-Fog" "Day-Night"
> images (internally called VIS++, as we use Visible, Near-IR and thermal
> IR channels to make them).  We also produce "Convective Diagnostic" and
> "Volcanic Ash" products by using the 10-bit data from the Unidata ADDE
> servers for those satellites.  You can see some of the products/regions
> that we do this for at:
> 
> http://wx.erau.edu/erau_sat/

Yup, I interacted with Fred some time ago about these products. At the
time, I asked if we could list ERAU as operating an open ADDE server,
and I was assured that I could.  The problem I found was the access
to the imagery was very slow...

re:
> Fred pointed out that the 2016 release of the McIDAS software can
> manipulate the new GOES-16 data, which I have confirmed after installing
> the release here and connecting to the preliminary data at the LEAD
> server.

Yes, Unidata McIDAS-x v2016a can be used to manipulate the ABI imagery
from GOES-16.  v2017 will be much better for doing this, however.

re:
> Question:  Are the data on your server any different from what we are
> getting via our NOAAPort downlink?

Yes and no.  Yes because we have the full set of GOES-16 imagery that
we are receiving on the GOES-R downlink that we installed on one of
the existing satellite pads up at the Mesa Lab.  No, because we are
also saving and serving the ABI imagery that is being sent in NOAAPort.

re:
> We seem to be getting the same
> regions, channels and times as I found on the Unidata ADDE server.

The imagery in NOAAPort is different than what is coming down in the
GOES ReBroadcast (GRB) in several ways:

- the Full Disk images in NOAAPort have been subsampled down to 6 km
  resolution while the imagery in the GRB has not

- the CONUS, PuertoRico and Mesoscale sectors have been remapped
  from their (semi) original Fixed Grid projection into a conical
  projection

  This results in a loss of information, and the Weather service
  recently ran a test of not remapping these into the conical
  projections.  I have not heard what the results of the test were.

- the calibration in the NOAAPort imagery is "dumbed down" from
  the calibration in the original GRB imagers

re:
> Our data are 12-bit, which is what Fred thought the Unidata data are
> as well.

The GRB imagery is 12 or 14 bit depending on the band (wavelength channel).

re:
> It is my (naive) assumption that the data we have here are the
> same as what you have.  Is this correct?

Again, yes and no.

re:
> (If so, then we won't have to
> transfer data from Unidata to ERAU for the Goes-16 part of Fred's images.)

We are preparing to be able to distribute full GRB imagery in the IDD.
The volume is slightly less than the full CONDUIT datastream, so it
is likely that a number of sites will be able to REQUEST and process
it.

re:
> Question:  From what I read, the current release of McIDAS is not able
> to serve the GOES-16 data, but this may or may not be true.  Is this
> the case, and if it is not able to serve the data, when will a release
> be available that will be able to serve the data?

The current Unidata release does not have enough of the needed code mods
to serve the data like it should.  The next version, v2017, will have all
of SSEC's code mods up through the version they released yesterday, v2017.2.

re:
> Our goal is to begin making multi-channel products with GOES-16 beginning
> next week, so that we can prepare for the operational transition of the
> satellite later this fall.  If there is a way to insert our NOAAPort data
> into the ADDE server here, that would be great.  If that is the case,
> could you please help with some instructions on how to do this?

The first thing that needs to happen is for the broadcast tiles to be
stitched together into full scenes and then the full scenes (images)
be written to disk in some sort of an organized fashion.

Question: are you stitching the NOAAport-delivered tiles together yet?
If not, you can use the ldm-alchemy package that Ryan May (Unidata)
put together and is making available in the Unidata section of GitHub.

re:
> Related to this, we also want to include the other data we get via
> NOAAPort in our ADDE collection.  For example, we have the following
> data on our disk under the Gempak data tree.  These are all stored in
> AREA files.
> 
> From the NOAAPort downlink, we have operational images from GOES-East and
> GOES-West (not shown here).
> 
> /opt/wxdata/gempak/images/sat/EAST-CONUS
> /opt/wxdata/gempak/images/sat/EAST-CONUS/4km
> /opt/wxdata/gempak/images/sat/EAST-CONUS/4km/WV
> /opt/wxdata/gempak/images/sat/EAST-CONUS/4km/IR
> /opt/wxdata/gempak/images/sat/EAST-CONUS/4km/3.9
> /opt/wxdata/gempak/images/sat/EAST-CONUS/8km
> /opt/wxdata/gempak/images/sat/EAST-CONUS/8km/13.3
> /opt/wxdata/gempak/images/sat/EAST-CONUS/1km
> /opt/wxdata/gempak/images/sat/EAST-CONUS/1km/VIS

Just so you know, the GOES-East/West imagery in NOAAPort are in GINI format,
not AREA.  Unidata McIDAS has a server for the imagery in GINI format, so
this is a minor point, but I felt like I had to make it anyway.

re:
> From Fred's code, we have a number of different satellite products, using
> just Goes-West, a CONUS view that is a mosaic of the two satellites,
> and a mosaic that includes the EUMETSAT data that we can get every
> 3 hours.  Currently these are made from the 10-bit GOES data that we
> can get from your ADDE servers, as well as the NCEP public ADDE server
> for the EUMETSAT products.
> 
> /opt/wxdata/gempak/images/sat/ERAU
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/VIS++
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/IR
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/WV
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/GCD
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/GCH
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/CLD
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/V1
> /opt/wxdata/gempak/images/sat/ERAU/WCONUS/TSRA
> 
> /opt/wxdata/gempak/images/sat/ERAU/CONUS
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/VIS++
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/IR
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/WV
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/TSRA
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/GCH
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/CLD
> /opt/wxdata/gempak/images/sat/ERAU/CONUS/V1
> 
> /opt/wxdata/gempak/images/sat/ERAU/WHEM
> /opt/wxdata/gempak/images/sat/ERAU/WHEM/IR
> /opt/wxdata/gempak/images/sat/ERAU/WHEM/VIS++
> /opt/wxdata/gempak/images/sat/ERAU/WHEM/WV

OK.

re:
> Is there an "easy" way for me to populate our ADDE server with the
> contents of the above directories?

Sure.  If you are running McIDAS on a system that has read access to the
data files you are referring to, the process is one of creating ADDE
dataset definitions that "point" to the files on disk.  It is that
simple.

re:
> I started to read the User Guide on
> this, but I got a bit confused.  Our ADDE server won't be responsible for
> writing the data to disk, unless it has to.

ADDE servers typically do not write to disk, but they can.  The only
time they write to disk is when one copies (IMGCOPY) an image from
one dataset to a second dataset.  The output dataset will be in AREA
format while the input one can be in a variety of formats.

re:
> We currently build these
> products on one virtual machine, and then we use pqinsert to push the
> products to our LDM that processes the Gempak data for NMAP and GARP
> software packages.  If possible, the ADDE server will be reading a
> read-only NFS mount.

That is all that is needed.

re:
> Request: If you could give me an example of how
> to add (at least) one of these paths to our ADDE server, I should be
> able to figure out the rest.

I will send a follow up email with a step-by-step as soon as you send
me a detailed listing of the contents of one of the directories you
listed above ... I want to see how you are naming the files.

re:
> Next, among the comments in our equipment grant review was a suggestion
> that we talk with you folks about what disks to buy and how to configure
> our servers on this end.  I am very happy to say that the networking
> improvements that were anticipated when we submitted the proposal have all
> been done, and a network bottleneck inside of our servers is not likely
> at this time.  Robert can give details of the infrastructure as desired.
> He is our system administrator, but we have to share him with other labs,
> which is why I am helping with things like building the McIDAS software
> and configuring the server.

OK.  Given appropriate login credentials, I can logon and help you out
also.

re:
> Request: We would like to request a conference call for some day next
> week, if you have the people and time available.  We are available
> any day after 3:00 pm EDT, which is 1:00 pm your time.  If any day
> will work for you, just let us know.  If you need an earlier time,
> then Tuesday and Thursday are our best options on this end.

I'd really rather not do conference calls if at all possible as there
is no "paper trail" that others can benefit from.  If a conference call
is the only way to get certain questions answered, then I will acquiesce.
Monday, Tuesday and Thursday are the best days for me next week.

re:
> Page two
> of the attached file provides the feedback from our proposal review,
> with the suggestion that we discuss the server configuration(s) before
> we proceed with any purchases.

OK.  We will have to take a look before commenting.

re:
> We are hoping to have as much as is possible of our infrastructure
> upgrades/configurations as we can before GOES-16 goes operational later
> this fall.

OK.  That is what Fred said also.

re:
> Thanks in advance for any help you can provide!

No worries.

re:
> Best wishes to all.  I hope to get out to Boulder some time to visit.

We'll be here :-)

re:
> In the meantime, maybe we can have a regional training workshop in the
> spring (while it is still winter in Boulder).  We do have a nice beach
> here; wink, wink.  (-:

I will pass this along.  You know, of course, that there are certain
responsibilities that go along with being the host site of a regional
workshop...

re:
> PS  The new ADDE server will answer to the old wxdata.db.erau.edu address
> that was once active.  We lost that in a combination hardware failure and
> staffing change that came to close in time to each other to recover from.

OK.

I'll be looking for a detailed listing of one of the directories of images
that you want your ADDE server to support.  Again, if you gave me login
access and some guidance of what you want accomplished, I could help
you out.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: HDN-934891
Department: Support McIDAS
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.