[IDV #QZB-369254]: A catalog-friendly compressed data format?

Hi Brian,

A compressed format that can be used for both thredds catalogs and the
IDV is netcdf4. In fact, netcdf4 is one of the two distribution formats for
GOES-R products.

Cheers,

Sean

> Hello Unidata.
> 
> As one of the showcases of our new Unidata server, a scientist here
> wants to offer high resolution precipitation estimates
> (derived from satellite input sources).
> 
> These are huge datasets, but have a lot of zero values.
> That community uses compressed formats for that reason.
> But I would like it to be IDV-accessible.
> 
> Is there a compressed CDM format?
> Or is this what the "satellite image" formats are good for,
> and are quantitative data readily convertible to such formats?
> 
> Any advice welcomed.
> 
> Thanks,
> Brian Mapes
> 
> 
> Begin forwarded message:
> 
> From: Robert Joyce - NOAA Affiliate <address@hidden<mailto:address@hidden>>
> Date: November 22, 2013 3:07:48 PM GMT-05:00
> To: Brian Mapes <address@hidden<mailto:address@hidden>>
> Subject: Re: Collecting Datasets for the new Unidata server
> 
> Hi Brian;
> 
> In a week or two CPC will stage it's first gauge corrected CMORPH 
> precipitation, I think at least a year of that would be a good candidate.  I 
> need to find out from Pingping what the format will be, then I can get a 
> number on the disk space usage.  Unfortunately we store CMORPH online in 
> monthly tar files of bzipped binary files.
> 
> Bob
> 
> 
> On Sun, Nov 10, 2013 at 3:56 PM, Brian Mapes 
> <address@hidden<mailto:address@hidden>> wrote:
> Hello MPO big obs dataset users.
> 
> (please forward to others I forgot freely, I am reply-alling an old
> vis email, and adding a few folks randomly, incompletely).
> 
> 
> Our new Unidata server is here, getting set up this week, with
> many TB of space for an OpenDAP-accessible data repository.
> 
> 
> Overall, I want to really try to use this sytsem to unify analyses of
> 
> 1. Current weather data (1-few months rolling archive),
> (1.5 a Forecasts archive for verification and predictability work?)
> 2. Climate context (summary stats from historical archives).
> 3. Specialized obs datasets to put in all this context (DYNAMO, CAROB, ?).
> 
> Things that exist online elsewhere, we will add virtual links to.
> 
> Hosting model-generated simulation output, on non-obs-aligned
> model timelines, is not the main purpose, but ask me case by case.
> 
> 
> For 1., Unidata will soon help me set up the LDM for weather feeds,
> including some backlogged verbal requests from several of you.
> The interested should email me - let's meet, after we get an
> initial configuration in place, and brainstorm improvements from there.
> 
> A Webpage harvester can be easily set up for things not in the LDM
> (e.g. is there a TIGGE site we could grab some forecast data from?)
> 
> 
> For 2. I'm looking to build up a Web-served collection of
> climatology statistic files, for all the common obs/analyses sets.
> So I can make anomaly and standardized anomaly mask layers
> for weather display templates. For all our favorite datasets,
> I'll need
> 
> * mean climo (at monthly, daily-mean, 3hourly resolutions) to make anomalies
> * stdev climo (at ",",") for making standardized anomaly (Z-score, t-test) 
> overlays
> 
> ? (other moments, or extremal range for declaring "unprecedented" events,
> or ??, if you can dream it let's do it)
> 
> 
> 
> I don't mind a little redundancy with vis cluster datasets, since
> this will be Web-open, and vis is a walled garden where
> big computations need fast access... so,
> Vis users, can someone point me to what's available there?
> I could login and do ls commands but help is appreciated.
> 
> 
> 
> REANALYSIS AND SST:
> Some are already out there.
> 
> PSD has NCEP1 and 20Cv2, and I made .ncml variable aggregations,
> but I don't have the climatogical stdev files for normalizing anomalies.
> 
> SST sets are out on servers pretty well but again rarely the stdev 
> climatology.
> 
> MERRA is wonderfully aggregated on a GDS server
> (which can even be harnessed to do the climo calculations
> for us I think, if one of you knows GDS syntax).
> 
> CFSR is maddeningly non-aggregated in the NOMADS access.
> If any of you have reorganized sets of CFS data I'd be interested.
> 
> 
> 
> SATELLITE AND RADAR DIRECT OBS:
> For satellite texture, NOAA IR and COLA TRMM3B42 archives are good,
> but sometimes as much as a couple years behind current. Again no stdev climo.
> 
> Bob Joyce will make us some unique high-res satellite precip products. 
> (right?)
> 
> I'd be happy to set up a harvester from any realtime-data sites, if some of 
> you
> know such sites let me know. (for things not in the LDM stream). Brian McN,
> know any radar-imagery archives? (ash.mmm.ucar.edu<http://ash.mmm.ucar.edu/> 
> would be great to
> have digitally as an archive for IDV layering, but McIDAS type headers needed)
> 
> 
> 
> Other ideas welcome too.
> Please feel free to be ACTIVIST!
> I am busy too so things take awhile if left to me.
> 
> Please forward this freely, I'm replying to an old email
> plus a few add-ons, forgetting many I'm sure.
> 
> Brian
> 
> 
> 
> On Oct 15, 2010, at 10:54 AM, Pedro DiNezio wrote:
> 
> 
> i could help with CMIP3, though i'm across the street. Katinka could help me.
> 
> over these years i've been able to build a nice database with IPCC AR4 
> experiments, including some that are not at the CMIP3 database, such as CCSM3 
> and GFDL models and some other files that i've got directly from the 
> modelling centers.
> 
> my database is quite consistent. what i would do is copying my database from 
> AOML to the RSMAS machine on a disk. it's about 8 to 10 Tb. i have a 2 Tb 
> portable disk, does anybody have larger disk?
> 
> i also have a beautiful set of matlab functions that read the (CMIP3 
> CMORized) netcdf files into matrices. these functions take care of the 
> calendars and the fact that some variables could be located in more than one 
> netcdf file. all this stays under "the hood" for the user. i'm sure they will 
> work for CMIP5. i would like to pass them to RSMAS students before i leave.
> 
> are you including observational datasets, such as:
> 
> HadSLP
> HadSST
> EN3
> ERSST
> ERSLP
> 
> ?
> 
> 
> -------- Original Message --------
> Subject:        Fwd: data sets
> Date:   Thu, 14 Oct 2010 21:28:46 -0400
> From:   Amy Clement <address@hidden><mailto:address@hidden>
> To:     Pedro DiNezio <address@hidden><mailto:address@hidden>
> 
> 
> could you chime in on this?
> 
> -------- Original Message --------
> Subject:        data sets
> Date:   Thu, 14 Oct 2010 15:44:12 -0400
> From:   Brian Mapes <address@hidden><mailto:address@hidden>
> To:     address@hidden<mailto:address@hidden>
> CC:     Ben Kirtman <address@hidden><mailto:address@hidden>, shuyi chen 
> <address@hidden><mailto:address@hidden>, amy clement 
> <address@hidden><mailto:address@hidden>, Robert Burgman 
> <address@hidden><mailto:address@hidden>, chidong Zhang 
> <address@hidden><mailto:address@hidden>
> 
> 
> 
> Happy to coordinate.
> 
> Trimming the less likely suspects off the email traffic...
> 
> can I ask for an ls of what you all already have in the house?
> 
> May I (should I) make copies onto the new machine?
> Instead of starting a cold search and download task by Web.
> 
> Brian
> 
> 
> On Oct 14, 2010, at 3:23 PM, address@hidden<mailto:address@hidden> wrote:
> 
> > Great idea. Maybe individuals or groups could volunteer to be
> > sponsors of
> > particular data sets and take responsibility for archiving/updating
> > them?
> >
> >> Can we coordinate some baseline datasets on there?
> >>
> >> I bet some of you have much of this data lying around on
> >> existing machines I'm only dimly aware of.
> >>
> >>
> >> Suggestions
> >>
> >> Top level:
> >> monthlies/
> >> dailies/
> >> subdailies/
> >>
> >> Analyses:
> >> ERA40/
> >> ERAInt/
> >> NCEP/
> >> NCEP2/
> >> CFSRR/
> >> MERRA/
> >> (others?)
> >>
> >> Satellite & products:
> >> SSMI/
> >> TRMM/
> >> CMAP/
> >> OLR/
> >> CLAUS/
> >> ISCCP/
> >> ??more?
> >>
> >> Recent baseline model climatologies:
> >> AM2/
> >> CM2/
> >> CAM4/
> >> CAM5/
> >> CCSM3/
> >> CCSM4/
> >> CFS/
> >>
> >> Model archives:
> >> CMIP3//
> >> CMIP5/
> >>
> >> On Oct 13, 2010, at 1:59 PM, Ben Kirtman wrote:
> >>
> >>> Dear Co-Conspirators,
> >>>
> >>> The rsmas vis host is ready for business.  The access is
> >>> vis1.ccs.miami.edu<http://vis1.ccs.miami.edu/> and you login with your 
> >>> usual CCS userid and
> >>> password. Your data disk is /userid and group access is your
> >>> individual default group.  If folks want to allow more sharing we
> >>> can use other groups, if they exist,  or they can put in a request
> >>> for a new group by logging in to their account at
> >>> http://iweb.ccs.miami.edu:8080/account/login.htm
> >>>
> >>> Applications  are installed in /usr/local  or /usr/local/apps.
> >>>
> >>> If anyone needs assistance migrating data or has any questions or
> >>> problems please just let us know (Warner).
> >>>
> >>> I tried logging in and using matlab and grads and they worked fine,
> >>> although you will need to make sure the application you are
> >>> interested is in your path.
> >>>
> >>> Many thanks to the CCS team for getting this up and running!
> >>>
> >>> -Ben
> >>>
> >>>
> >>> --
> >>> Ben Kirtman
> >>> Division of Meteorology and Physical Oceanography
> >>> Rosenstiel School for Marine and Atmospheric Science
> >>>
> >>> and
> >>>
> >>> The Center for Computational Science
> >>> University of Miami
> >>>
> >>> 4600 Rickenbacker Causeway
> >>> Miami Fl 33149
> >>> Phone: 305-421-4046<tel:305-421-4046>
> >>> address@hidden<mailto:address@hidden>
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> Brian Mapes
> address@hidden<mailto:address@hidden> or address@hidden<mailto:address@hidden>
> 
> 
> 
> 
> 

Ticket Details
===================
Ticket ID: QZB-369254
Department: Support IDV
Priority: High
Status: Open