[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[NOAAPORT #LQO-967392]: [noaaport] Lack of GOES16 L2 products from HDS feed

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [NOAAPORT #LQO-967392]: [noaaport] Lack of GOES16 L2 products from HDS feed
  • Date: Tue, 03 Apr 2018 12:19:42 -0600
Hi Mike,

I drove your post to the noaaport and ldm-users email list into our inquiry
tracking system since we are not seeing the same thing as you.

re:
> I'm noticing that GOES-16 L2 Derived products appear to have largely gone
> missing from the HDS feed, mostly over NOAAPort, and I was wondering if
> anyone else had seen the same thing.
> 
> Based on some RTstats
> 
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+wxgoes.cod.edu
> 
> we've been getting a fairly regular amount of HDS data into our server,
> here seen mostly via NOAAPort, up until about a day ago.  Since then, very
> little if anything has been seen over NP, and only intermittent/irregular
> data is seen via Unidata feeds.  On this server, I'm limiting the HDS
> feed to "IXT.99 KNES" and "IXT[WXY]01 KNES" to just get the L2 products;
> those are based off the various change notices I've seen over time,
> and they've worked till now.

It is a bit hard to compare the real-time stats from wxgoes.cod.edu since
you are requesting a portion of the HDS datastream, but a direct comparison
can be made for your NOAAPort ingest machines and those that are feeding into
the IDD.  Volume plots for the full HDS content on the 5 NOAAPort ingest
machines that are feeding into the IDD show more or less the same volume
being ingested in the HDS feed for the past 3 days:

noaaport1.cod.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+noaaport1.cod.edu

noaaport2.cod.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+noaaport2.cod.edu

mistral.srcc.lsu.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+mistral.srcc.lsu.edu

np1.ssec.wisc.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+np1.ssec.wisc.edu

np2.ssec.wisc.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+np2.ssec.wisc.edu

chico.unidata.ucar.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+chico.unidata.ucar.edu

leno.unidata.ucar.edu HDS volume:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+leno.unidata.ucar.edu

The same volume is being made available on all of the real server backend
machines for the two toplevel IDD relay clusters that we operate here in
UCAR.  For example:

cluster0.unidata.ucar.edu HDS volume (a backend machine for 
iddb.unidata.ucar.edu):
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+cluster0.unidata.ucar.edu

uni14.unidata.ucar.edu HDS volume (a backend machine for idd.unidata.ucar.edu):
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+uni14.unidata.ucar.edu

re:
> Here is the RTstats graph from our NP server
> 
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?HDS+noaaport1.cod.edu
> 
> showing a substantial drop off in volume.  This server should be getting
> everything from the HDS feed.

Have you checked the ingest logs on your NOAAPort machines to see if there
has been a substantial increase in the number of errors being reported?

re:
> I went through the typical troubleshooting (e.g. del/remake queue, restart
> LDM) but I'm still not seeing those products via NP.

On which machine(s) did you do the various troubleshooting steps?  If it
was only on wxgoes.cod.edu, you are probably looking at the wrong machine(s).

re:
> Did something change a day ago, or could this be something else on my end?

We are not seeing the kind of drop in HDS volume on our end that you
are seeing on yours.

NB: I have no explanation for why you wouldn't be receiving all of the
HDS products via an IDD feed!  Did anything change on your end recently
(e.g., change in the ldmd.conf REQUEST, network bandwidth restrictions,
etc.)?

If you want to check on the L2 products we have ingested and are making
available via ADDE, you can "point" (DATALOC) at either of the following
machines:

lead.unidata.ucar.edu
atm.ucar.edu

The ADDE group is NPGOESR, and the various descriptors can be gleened
from the listing produced by DSINFO:

DATALOC ADD NPGOESR atm.ucar.edu
DSINFO IMAGE NPGOESR

        Dataset Names of Type: IMAGE in Group: NPGOESR

Name         NumPos   Content
------------ ------   --------------------------------------
AODCN        99999    GOES-16 Level2 CONUS Aerosol Optical Depth Product
AODFD        99999    GOES-16 Level2 Full Disk Aerosol Optical Depth Product
AODM1        99999    GOES-16 Level2 Mesoscale-1 Aerosol Optical Depth Product
AODM2        99999    GOES-16 Level2 Mesoscale-2 Aerosol Detection Optical Dep
ARSLCN       99999    GOES-16 Level2 CONUS Aerosol Detection Products
ARSLFD       99999    GOES-16 Level2 Full Disk Aerosol Detection Products
ARSLM1       99999    GOES-16 Level2 Mesoscale-1 Aerosol Detection Products
ARSLM2       99999    GOES-16 Level2 Mesoscale-2 Aerosol Detection Products
CMSKCN       99999    GOES-16 Level2 CONUS Cloud Mask Product
CMSKFD       99999    GOES-16 Level2 Full Disk Cloud Mask Product
CMSKM1       99999    GOES-16 Level2 Mesoscale-1 Cloud Mask Product
CMSKM2       99999    GOES-16 Level2 Mesoscale-2 Cloud Mask Product
CODCN        99999    GOES-16 Level2 CONUS Cloud Optical Depth Product
CODFD        99999    GOES-16 Level2 Full Disk Cloud Optical Depth Product
CODM1        99999    GOES-16 Level2 Mesoscale-1 Cloud Optical Depth Product
CODM2        99999    GOES-16 Level2 Mesoscale-2 Cloud Optical Depth Product
CONUSC01     99999    GOES-16 CONUS 0.47 um VIS aerosol-over-land
CONUSC02     99999    GOES-16 CONUS 0.64 um VIS clouds fog/insol/winds
CONUSC03     99999    GOES-16 CONUS 0.86 um Near IR veg/burn scar/aerosol
CONUSC04     99999    GOES-16 CONUS 1.37 um Near IR cirrus cloud
CONUSC05     99999    GOES-16 CONUS 1.61 um Near IR cloud phase/snow
CONUSC06     99999    GOES-16 CONUS 2.24 um Near IR land/cloud vege/snow
CONUSC07     99999    GOES-16 CONUS 3.89 um IR Sfc/cloud/fog/fire/winds
CONUSC08     99999    GOES-16 CONUS 6.17 um IR High-level WV/winds/rainfall
CONUSC09     99999    GOES-16 CONUS 6.93 um IR Mid-level WV/winds/rainfal
CONUSC10     99999    GOES-16 CONUS 7.34 um IR Lower-level WV/winds & SO2
CONUSC11     99999    GOES-16 CONUS 8.44 um IR Total WV cloud phase/dust
CONUSC12     99999    GOES-16 CONUS 9.61 um IR Total ozone/turbulence/wind
CONUSC13     99999    GOES-16 CONUS 10.3 um IR Surface & cloud
CONUSC14     99999    GOES-16 CONUS 11.2 um IR Imagery/SST/clouds/rainfall
CONUSC15     99999    GOES-16 CONUS 12.3 um IR Total water/ash and SST
CONUSC16     99999    GOES-16 CONUS 13.3 um IR Air temp/cloud hgt and amt
CPSCN        99999    GOES-16 Level2 CONUS Cloud Particle Size Product
CPSFD        99999    GOES-16 Level2 Full Disk Cloud Particle Size Product
CPSM1        99999    GOES-16 Level2 Mesoscale-1 Cloud Particle Size Product
CPSM2        99999    GOES-16 Level2 Mesoscale-2 Cloud Particle Size Product
CTHCN        99999    GOES-16 Level2 CONUS Cloud Top Height Product
CTHFD        99999    GOES-16 Level2 Full Disk Cloud Top Height Product
CTHM1        99999    GOES-16 Level2 Mesoscale-1 Cloud Top Height Product
CTHM2        99999    GOES-16 Level2 Mesoscale-2 Cloud Top Height Product
CTPHCN       99999    GOES-16 Level2 CONUS Cloud Top Phase Product
CTPHFD       99999    GOES-16 Level2 Full Disk Cloud Top Phase Product
CTPHM1       99999    GOES-16 Level2 Mesoscale-1 Cloud Top Phase Product
CTPHM2       99999    GOES-16 Level2 Mesoscale-2 Cloud Top Phase Product
CTPRCN       99999    GOES-16 Level2 CONUS Cloud Top Pressure Product
CTPRFD       99999    GOES-16 Level2 Full Disk Cloud Top Pressure Product
CTPRM1       99999    GOES-16 Level2 Mesoscale-1 Cloud Top Pressure Product
CTPRM2       99999    GOES-16 Level2 Mesoscale-2 Cloud Top Pressure Product
CTTCN        99999    GOES-16 Level2 CONUS Cloud Top Temperature Product
CTTFD        99999    GOES-16 Level2 Full Disk Cloud Top Temperature Product
CTTM1        99999    GOES-16 Level2 Mesoscale-1 Cloud Top Temperature Product
CTTM2        99999    GOES-16 Level2 Mesoscale-2 Cloud Top Temperature Product
DSICN        99999    GOES-16 Level2 CONUS Derived Stability Indices Products
DSIFD        99999    GOES-16 Level2 Full Disk Derived Stability Indices Produ
DSIM1        99999    GOES-16 Level2 Mesoscale-1 Derived Stability Indices Pro
DSIM2        99999    GOES-16 Level2 Mesoscale-2 Derived Stability Indices Pro
FDC01        99999    GOES-16 FullDisk 0.47 um VIS aerosol-over-land
FDC02        99999    GOES-16 FullDisk 0.64 um VIS clouds fog/insol/winds
FDC03        99999    GOES-16 FullDisk 0.86 um Near IR veg/burn scar/aerosol
FDC04        99999    GOES-16 FullDisk 1.37 um Near IR cirrus cloud
FDC05        99999    GOES-16 FullDisk 1.61 um Near IR cloud phase/snow
FDC06        99999    GOES-16 FullDisk 2.24 um Near IR land/cloud vege/snow
FDC07        99999    GOES-16 FullDisk 3.89 um IR Sfc/cloud/fog/fire/winds
FDC08        99999    GOES-16 FullDisk 6.17 um IR High-level WV/winds/rainfall
FDC09        99999    GOES-16 FullDisk 6.93 um IR Mid-level WV/winds/rainfal
FDC10        99999    GOES-16 FullDisk 7.34 um IR Lower-level WV/winds & SO2
FDC11        99999    GOES-16 FullDisk 8.44 um IR Total WV cloud phase/dust
FDC12        99999    GOES-16 FullDisk 9.61 um IR Total ozone/turbulence/wind
FDC13        99999    GOES-16 FullDisk 10.3 um IR Surface & cloud
FDC14        99999    GOES-16 FullDisk 11.2 um IR Imagery/SST/clouds/rainfall
FDC15        99999    GOES-16 FullDisk 12.3 um IR Total water/ash and SST
FDC16        99999    GOES-16 FullDisk 13.3 um IR Air temp/cloud hgt and amt
FHSCN        99999    GOES-16 Level2 CONUS Fire Hot Spot Products
FHSFD        99999    GOES-16 Level2 Full Disk Fire Hot Spot Products
FHSM1        99999    GOES-16 Level2 Mesoscale-1 Fire Hot Spot Products
FHSM2        99999    GOES-16 Level2 Mesoscale-2 Fire Hot Spot Products
LSTCN        99999    GOES-16 Level2 CONUS Land Surface Temperature Product
LSTFD        99999    GOES-16 Level2 Full Disk Land Surface Temperature Produc
LSTM1        99999    GOES-16 Level2 Mesoscale-1 Land Surface Temperature Prod
LSTM2        99999    GOES-16 Level2 Mesoscale-2 Land Surface Temperature Prod
M1C01        99999    GOES-16 Meso1 0.47 um VIS aerosol-over-land
M1C02        99999    GOES-16 Meso1 0.64 um VIS clouds fog/insol/winds
M1C03        99999    GOES-16 Meso1 0.86 um Near IR veg/burn scar/aerosol
M1C04        99999    GOES-16 Meso1 1.37 um Near IR cirrus cloud
M1C05        99999    GOES-16 Meso1 1.61 um Near IR cloud phase/snow
M1C06        99999    GOES-16 Meso1 2.24 um Near IR land/cloud vege/snow
M1C07        99999    GOES-16 Meso1 3.89 um IR Sfc/cloud/fog/fire/winds
M1C08        99999    GOES-16 Meso1 6.17 um IR High-level WV/winds/rainfall
M1C09        99999    GOES-16 Meso1 6.93 um IR Mid-level WV/winds/rainfal
M1C10        99999    GOES-16 Meso1 7.34 um IR Lower-level WV/winds & SO2
M1C11        99999    GOES-16 Meso1 8.44 um IR Total WV cloud phase/dust
M1C12        99999    GOES-16 Meso1 9.61 um IR Total ozone/turbulence/wind
M1C13        99999    GOES-16 Meso1 10.3 um IR Surface & cloud
M1C14        99999    GOES-16 Meso1 11.2 um IR Imagery/SST/clouds/rainfall
M1C15        99999    GOES-16 Meso1 12.3 um IR Total water/ash and SST
M1C16        99999    GOES-16 Meso1 13.3 um IR Air temp/cloud hgt and amt
M2C01        99999    GOES-16 Meso2 0.47 um VIS aerosol-over-land
M2C02        99999    GOES-16 Meso2 0.64 um VIS clouds fog/insol/winds
M2C03        99999    GOES-16 Meso2 0.86 um Near IR veg/burn scar/aerosol
M2C04        99999    GOES-16 Meso2 1.37 um Near IR cirrus cloud
M2C05        99999    GOES-16 Meso2 1.61 um Near IR cloud phase/snow
M2C06        99999    GOES-16 Meso2 2.24 um Near IR land/cloud vege/snow
M2C07        99999    GOES-16 Meso2 3.89 um IR Sfc/cloud/fog/fire/winds
M2C08        99999    GOES-16 Meso2 6.17 um IR High-level WV/winds/rainfall
M2C09        99999    GOES-16 Meso2 6.93 um IR Mid-level WV/winds/rainfal
M2C10        99999    GOES-16 Meso2 7.34 um IR Lower-level WV/winds & SO2
M2C11        99999    GOES-16 Meso2 8.44 um IR Total WV cloud phase/dust
M2C12        99999    GOES-16 Meso2 9.61 um IR Total ozone/turbulence/wind
M2C13        99999    GOES-16 Meso2 10.3 um IR Surface & cloud
M2C14        99999    GOES-16 Meso2 11.2 um IR Imagery/SST/clouds/rainfall
M2C15        99999    GOES-16 Meso2 12.3 um IR Total water/ash and SST
M2C16        99999    GOES-16 Meso2 13.3 um IR Air temp/cloud hgt and amt
PRC01        99999    GOES-16 PuertoRico 0.47 um VIS aerosol-over-land
PRC02        99999    GOES-16 PuertoRico 0.64 um VIS clouds fog/insol/winds
PRC03        99999    GOES-16 PuertoRico 0.86 um Near IR veg/burn scar/aerosol
PRC04        99999    GOES-16 PuertoRico 1.37 um Near IR cirrus cloud
PRC05        99999    GOES-16 PuertoRico 1.61 um Near IR cloud phase/snow
PRC06        99999    GOES-16 PuertoRico 2.24 um Near IR land/cloud vege/snow
PRC07        99999    GOES-16 PuertoRico 3.89 um IR Sfc/cloud/fog/fire/winds
PRC08        99999    GOES-16 PuertoRico 6.17 um IR High-level WV/winds/rainfa
PRC09        99999    GOES-16 PuertoRico 6.93 um IR Mid-level WV/winds/rainfal
PRC10        99999    GOES-16 PuertoRico 7.34 um IR Lower-level WV/winds & SO2
PRC11        99999    GOES-16 PuertoRico 8.44 um IR Total WV cloud phase/dust
PRC12        99999    GOES-16 PuertoRico 9.61 um IR Total ozone/turbulence/win
PRC13        99999    GOES-16 PuertoRico 10.3 um IR Surface & cloud
PRC14        99999    GOES-16 PuertoRico 11.2 um IR Imagery/SST/clouds/rainfal
PRC15        99999    GOES-16 PuertoRico 12.3 um IR Total water/ash and SST
PRC16        99999    GOES-16 PuertoRico 13.3 um IR Air temp/cloud hgt and amt
TPWCN        99999    GOES-16 Level2 CONUS Total Precipitable Water Product
TPWFD        99999    GOES-16 Level2 Full Disk Total Precipitable Water Produc
TPWM1        99999    GOES-16 Level2 Mesoscale-1 Total Precipitable Water Prod
TPWM2        99999    GOES-16 Level2 Mesoscale-2 Total Precipitable Water Prod

DSINFO -- done

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: LQO-967392
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.