[McIDAS #CLT-967906]: update and some questions

Hi Ioan,

re:
> long time no hear...

Yes indeed!

re:
> how are you doing....? have you been on holiday? We had a weird summer in
> Europe this year,

I'm doing well, thanks.  I did have a holiday in England in July, but I've
been back for almost a month now.

re:
> the June and July were quite cold.....  but now the temps are as they
> should.

It has been very warm here, and WAY too dry!

re:
> As for me no real  holiday this year ...I was working:)))

I know the feeling :-(

re:
> I have managed to port pyadde and pyarea to python 3 (async) and built an
> integrated (multisource) data downloader which should ensure we do not
> miss any data in case some sources do not work.

This sounds very interesting and useful!

re:
> I'll send you the updated libs (pyadde pyarea) once I package them.

Thanks.

re:
> I want to discuss shortly two things:
> 
> A. GOES16
> I believe I have already asked you this question but I am going to do it
> again in case you got some update on the matter in the meantime.
> 
> Right now we have 3 sources of GOES16 data
> 
> 1. real time 
> https://www.googleapis.com/storage/v1/b/gcp-public-data-goes-16/o NetCDF
> 2. lead.unidata.ucar.edu McIDAS AREA
> 3. CLASS NetCDF

Excellent, I always like redundancy.

You can add to your list:

4. atm.ucar.edu - McIDAS AREA (via ADDE)
5. thredds-test.unidata.ucar.edu netCDF (via the TDS)

Coming up sometime reasonably soon will be:

6. adde.ucar.edu - McIDAS AREA (via ADDE)
7. thredds.ucar.edu - netCDF (via the TDS)

re:
> I was wondering if you got any updates/news from SSEC on the ABI McIDASX
> server because some calibration parameters are still missing from the AREA
> files.

I don't recall you asking this previously, so please forgive the questions:

- which calibration parameters are missing from the calibration block in a
  McIDAS AREA file?

I just looked at the SSEC-written server code for the GRB-delivered ABI imagery,
and I think that it reads and stores all of the calibration information that
is available in the netCDF-4 files that are created by the CSPP GEO software
that we are running on our GOES-16 downlink.

Here are what I deem to be calibration-related values in the GRB netCDF-4 files
(please correct me when I am wrong/missing something!):

        short Rad(y, x) ;
                Rad:_FillValue = 1023s ;
                Rad:long_name = "ABI L1b Radiances" ;
                Rad:standard_name = "toa_outgoing_radiance_per_unit_wavelength" 
;
                Rad:_Unsigned = "true" ;
                Rad:sensor_band_bit_depth = 10b ;
                Rad:valid_range = 0s, 1022s ;
                Rad:scale_factor = 0.8121064f ;
                Rad:add_offset = -25.93665f ;
                Rad:units = "W m-2 sr-1 um-1" ;
                Rad:resolution = "y: 0.000028 rad x: 0.000028 rad" ;
                Rad:coordinates = "band_id band_wavelength t y x" ;
                Rad:grid_mapping = "goes_imager_projection" ;
                Rad:cell_methods = "t: point area: point" ;
                Rad:ancillary_variables = "DQF" ;

The operative calibration values from this list should be: scale_factor and 
add_offset.

        float band_wavelength(band) ;
                band_wavelength:long_name = "ABI band central wavelength" ;
                band_wavelength:standard_name = 
"sensor_band_central_radiation_wavelength" ;
                band_wavelength:units = "um" ;

The operative calibration value here is the band_wavelength itself.

        float kappa0 ;
                kappa0:_FillValue = -999.f ;
                kappa0:long_name = "Inverse of the incoming top of atmosphere 
radiance at current earth-sun distance (PI d2 esun-1)-1, where d is the ratio 
of instantaneous Earth-Sun distance divided by the mean Earth-Sun distance, 
esun is the bandpass-weighted solar irradiance and PI is a standard constant 
used to convert ABI L1b radiance to reflectance" ;
                kappa0:units = "(W m-2 um-1)-1" ;
                kappa0:coordinates = "band_id band_wavelength t" ;
                kappa0:cell_methods = "t: mean" ;
        float planck_fk1 ;
                planck_fk1:_FillValue = -999.f ;
                planck_fk1:long_name = "wavenumber-dependent coefficient (2 h 
c2/ nu3) used in the ABI emissive band monochromatic brightness temperature 
computation, where nu =central wavenumber and h and c are standard constants" ;
                planck_fk1:units = "W m-1" ;
                planck_fk1:coordinates = "band_id band_wavelength" ;
        float planck_fk2 ;
                planck_fk2:_FillValue = -999.f ;
                planck_fk2:long_name = "wavenumber-dependent coefficient (h c 
nu/b) used in the ABI emissive band monochromatic brightness temperature 
computation, where nu = central wavenumber and h, c, and b are standard 
constants" ;
                planck_fk2:units = "K" ;
                planck_fk2:coordinates = "band_id band_wavelength" ;
        float planck_bc1 ;
                planck_bc1:_FillValue = -999.f ;
                planck_bc1:long_name = "spectral bandpass correction offset for 
brightness temperature (B(nu) - bc_1)/bc_2 where B()=planck_function() and 
nu=wavenumber" ;
                planck_bc1:units = "K" ;
                planck_bc1:coordinates = "band_id band_wavelength" ;
        float planck_bc2 ;
                planck_bc2:_FillValue = -999.f ;
                planck_bc2:long_name = "spectral bandpass correction scale 
factor for brightness temperature (B(nu) - bc_1)/bc_2 where 
B()=planck_function() and nu=wavenumber" ;
                planck_bc2:units = "1" ;
                planck_bc2:coordinates = "band_id band_wavelength" ;

All of these values are needed for calibration.

The McIDAS-X server code for GRB ABI imagery contains:

      memcpy( (void *)&cal_block[0], (void *)"ABIN",4); 
      cal_block[j]    = image_band;
      cal_block[j+1]  = (int)(WaveLength[image_band-1] *10000000.0);
      cal_block[j+2]  = (int)(VAR_scale[image_band-1]  *10000000.0); 
      cal_block[j+3]  = (int)(VAR_offset[image_band-1] *10000000.0);
      cal_block[j+4]  = 0;
      cal_block[j+5]  = 0;
      cal_block[j+6]  = 0;
      cal_block[j+7]  = 0;
      cal_block[j+8]  = 0;
      cal_block[j+9]  = 0; 
      cal_block[j+10] = 0;
      cal_block[j+11] = (int)(kappa[image_band-1]      *10000000.0);
      cal_block[j+12] = 0;
      cal_block[j+13] = (int)(planckFK1[image_band-1]  *10000.0);
      cal_block[j+14] = (int)(planckFK2[image_band-1]  *10000.0);
      cal_block[j+15] = (int)(planckBC1[image_band-1]  *10000000.0);
      cal_block[j+16] = (int)(planckBC2[image_band-1]  *10000000.0);
      cal_block[j+17] = VAR_FillValue[image_band-1];

The association between the values I listed from the netCDF-4 files
are:

netCDF-4          AREA calibration block
----------------+-----------------------
band_wavelength  Wavelength
scale_factor     VAR_scale
add_offset       VAR_offset
kappa0           kappa
planck_fk1       planckFK1
planck_fk2       planckFK2
planck_bc1       planckBC1
planck_bc2       planckBC2

re:
> I have ways to deal with it but It would be so nice if the AREA
> files that come from lead would be one to one  with the NetCDF files.

Which calibration value(s) are missing?

re:
> B: the rebuild of GOES archives using pygvar
> 
> I have made some nice progress with this task. However my boss, is not 100%
> happy because the GOES image are of course a little wobbly (+- 1 px) mostly
> the VIS channel.

I assume that you mean the GOES GVAR images.  I see that the animated GIFs
that you attached were all from GOES-8...

re:
> I am attaching 3 files so you can have a look.
> My question is:...do you know anybody /somebody who tried to correct these
> kind of issues with GOES satellites

I know that a LOT of work was done to correct the inter-band registration
for IR channels for GOES GVAR satellites.  There are more than one published
papers on what the problem was and how it can be corrected.

I feel certain that there has been a LOT of work done on correcting the
registration for older GOES satellites (e.g., GOES-8), but I can't give
you any references, sorry.

re:
> pygvar does a good enough job in my opinion but that one pixel gives my
> boss trouble.:))).

I have observed registration problems with GOES GVAR AND GRB images.  The
only good news is that I see far fewer in the GRB images than I saw in
GVAR images.

re:
> All the best

You too!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CLT-967906
Department: Support McIDAS
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.