Bob Simons wrote:
John Caron wrote:Hi Bob:We've been working on these kinds of problems, and thinking about how/whether to use sequences or not. So im just going to work through your use case below:Bob Simons wrote:I am trying to figure out how to store lots of sequence-like data in .nc files for efficient access via OPeNDAP. In particular, I am trying to determine if actual OPeNDAP Sequences (Structures with an unlimited dimension in the .nc file) is not appropriate for our purposes.Yes, I could store the data in a file on the computer where the program needing access is running, and not have to access it via OPeNDAP, so that network transmission time would be minimized. But this project is partly an experiment in dealing with remotely accessed data. So I am trying to design a solution where the data is accessed from another computer via OPeNDAP.Here's an example. Let's say I want to store all NDBC buoy data in a .nc file. There are over 100 buoys. For each buoy, there are readings for some time period (e.g., just 1989, or from 1990 to the present). The readings are an hour apart. Several variables (e.g., WindSpeed and WindDirection) are measured at each time point. Since we work with real-time data, I plan to update this file frequently (every day, but ideally every hour).How large do you expect the file to get (total number of readings ) ? reading == record == one structure in the sequence.Approximately:There are 400 buoys * ~8 years of data * 8760 hours/year = ~28,000,000 records.Given the great variation to time ranges for each buoy, I will probably arrange it as 400 sequences (one per buoy), with an average of 8 * 8760 = 70,080 records per sequence.Each record has 1 double and 15 floats; hence 68 bytes/record * 28,000,000 records = ~1.9GBThe problem is, I need to have *quick* access via OPeNDAP:* Across all buoys at a specific time point, e.g., What is the wind speed at all buoys at 2004-12-14T09:00Z? * Or, for all time points available, what is the wind speed, for example, at a specific buoy?Do you need other queries, like "find all readings with wind speed > 30 mph" ??In general, no.The most common variant of the first request above is: restrict the request to all buoys in a rectangular geographic region, .... But I can separately manage and subset the geographic locations of the buoys, if needed.Regarding the first requirement, from what I understand, if I use sequences, there is no way to get the data for a given time point without reading either the whole file up to that time point, or without reading a whole variable. Either of which would seem to take too long if I want the values for 100 buoys (given that I am using OPeNDAP to connect to a remote computer and want the response quickly for my CoastWatch Browser program, which graphs the data for on-line users who want a quick response).im not sure how your specific dods server works, but theres a good change that the server has to read the entire file to answer your query. We need to find that out, if we are going to figure out how to scale this.I definitely want to avoid having the server read the entire file for each query. That's why I ask (below) about avoiding structures and just using lots of variables.in the opendap world, you can in fact put a CE (constraint expression) on the sequence, eg your first query would be something like "time = 2004-12-14T09:00Z", and your second "buoy=2309" (ill have to check the exacct syntax). Now we dont yet properly support that in the nj22 library, but I think it may not be that hard to do. The hard problem is probably on the server, if it has to read the entire file to answer it.I'm not sure why you say it has to read the entire file. If I set up the file in certain ways, can't that be avoided?
it depends on the server.
Since the time range of available data for each buoy varies greatly, it seems grossly wasteful of space to have a common Time dimension for all buoys. Doing so would probably force me over the 2GB file size, which is generally trouble. So I am thinking about either: * A time dimension for each buoy (e.g., time14978 for buoy 14978) and a several variables which use that dimension to store the data for that buoy (e.g., windSpeed14978, windDirection14978, etc.). This setup would be replicated for each buoy.I am leaning toward this. It is easy to understand. It doesn't use any special features of .nc or OPeNDAP, so should work will different servers and clients.* Or, a Group for each buoy, again with a time dimension and several variables in each group to store the data for each buoy. (If this is a new .nc feature, does OPeNDAP deal with this yet?) * Or, an ArrayObject.1D of variables, each element of which is an ArrayObject.1D of the variables for a given buoy. (I'm not sure if this can be done.) * Or, an ArrayObject.2D of variables, with buoys as one dimension and the various variables (e.g., WindSpeed, WindDirection) on the other dimension. (I'm not sure if this can be done.)Our current thinking on how to write netcdf files for "observation data" is written up at:http://www.unidata.ucar.edu/software/netcdf-java/formats/UnidataObsConvention.htmlin particular, appending records using backwards-linked lists seems like a good solution, and its what we are currently doing with the realtime metar data on motherlode.I really don't like linked lists. They force each query to go through the all the rows (and read all of the data for each row).I think separate variables are the way to go. They can be made expandable in the way that Java's ArrayList is expandable: have a backing array, and keep track of how many elements are currently in use (size). If you need more capacity, make a new larger array and copy the values to it. Then you can get random access to any value in any variable. Further, if you need to do constraints, you only need to read the constraint variables, and even then you can minimize the reads. For example, if I sort an buoy's records by time and have a query like "time >= t1 && time <= t2 && windSpeed > 30", I don't even have to read the windSpeed variable until I find a record in the correct time range. And I never have to read the other variables until I know the constraint expression is satisfied.In fact, part of my raising this question was to try to figure out why/when structures/sequences are a good approach. I feel like I'm missing something. It looks like they are implemented as linked lists. If so, they don't look like a good data structure to me, because they force all file accesses to go through the whole file. They are efficient when appending data (which you do infrequently and when you care less about speed), but inefficient for searches (both sequential searches of one or a few variables, or random access to any datum) (which you do frequently and when you really care about speed). And there are other data structures (in the style of Java's ArrayList) which are efficient for writing (random access or appending) and reading (sequential or random access). Comments?
Im a bit confused if were talking about the server or the client?Im assuming that you want to write a netcdf file that a opendap server can serve? I dont know of any that would automatically serve sequences. We are looking at adding that to the THREDDS data server (TDS), but havent yet.
Netcdf-3 files can only be expanded along one dimension. Netcdf-4 wont have that limitation, but they arent ready yet. In the UnidataObsConvention, we write along the record dimension as the data comes in, there's a seperate linked list for each station. As long as you keep the files moderately small, this isnt a bad solution for modest datasets. We may add an external indexing capability in the TDS for large datasets.To optimize the search, I would probably not use the linked list, but the "contiguous list" option (see the UnidataObsConvention.html). You definitetly want to use the record dimension, by the way, you could see a factor of 100 times slower without it. You could have the current data come into a daily file (using linked lists?), then add the daily file to the archive, with periodic rewriting of the file.
But none of these are options unless the server can deal with it. So these are just some ideas that I am considering for the TDS server.Im afraid Ive run out of time to go into more depth. We'll have to continue this after the holidays.
Happy Holidays!
I plan to solve the updating problem by leaving rows of missing values at the end of the data for each active buoy. As new data comes in, I will replace the missing values with actual data. Then, I only have to rewrite the file (to add more rows of missing values) once in a while, not every time.the above approach, if it works for you, probably obviates this.Which approach sounds best? Is there another approach? Do you have any advice?Are sequences the wrong way to go? Of course, that could change if one could efficiently access specific ranges from variables in a Sequence/Structure. But it my understanding that that is not currently possible.The DAP 2 spec currently does not allow this. But the whole point of sequences is to allow you to subset using a query (i think its called a selection), and then only return the data needed, so you dont need index subsetting.But if the server has to go through the whole file, it will never be fast.Although I gave this specific example, we store a lot of sequence-like data where I work. Whatever .nc file structure is appropriate for the buoys will likely be appropriate for much of this other data. So I want to get it right.Right now, Id say that it depends on what server you are using. Sequences are elegant, but they are a different animal from indexed access that is the bread and butter of netcdf files.The critical things to answer first: 1. How many records will you serve? What about in the future?Approximately: 400 buoys * ~8 years of data * 8760 hours/year = ~28,000,000 records Each record has 1 double and 15 floats; hence 68 bytes/record * 28,000,000 records = ~1.9GB More than half of the buoys are active so it will grow by about: 300 buoys * 8760 hours/year = 2,628,000 records / year (almost 200MB/year)Given that it is close to 2GB, I may separate it into a file for inactive buoys and a file for active buoys.2. What queries do you need to support? 3. What response time is acceptable ?I would like 1 second search time on the server + whatever the network transmission time is for OPeNDAP to send me the results. Note that the results are often/usually < 100 KB of data. I am willing to do a lot to get that response time, e.g., store the buoy locations and time ranges in memory. Buoy readings are every hour, but with gaps. So perhaps I would constrain the times for each buoy's reading to be regularly spaced (e.g., missing data would appear as rows of missing values in the file), so that I can very quickly calculate the relevant row(s) of data based on time constraints.4. What clients do you want to support? Just your own, or more general?I guess I only care about my client. But it seems like if I do this right, it will be useful to any client that works with a given server. For me, OPeNDAP is here now and available for no effort on my part. So, any OPeNDAP client can use the .nc file. Presumable, other servers (LAS, THREDDS) could use the file, too, in the future.5. What server do you want to use? Does it matter?It doesn't matter to me, except for ease of use. So I'm strongly inclined to use one of the OPeNDAP servers which is already administered here (by someone else). I'll make the file. They'll serve it.Sincerely, Bob Simons Satellite Data Product Manager Environmental Research Division NOAA Southwest Fisheries Science Center 1352 Lighthouse Ave Pacific Grove, CA 93950-2079 (831)658-3205 address@hidden <>< <>< <>< <>< <>< <>< <>< <>< <><