[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [support] Re: Could not read vdata records
- Subject: Re: [support] Re: Could not read vdata records
- Date: Tue, 27 Sep 2011 07:59:45 -0600
Tony,
I'm traveling this week but I'll try to get back to you abut this stuff...
James
On Sep 26, 2011, at 10:11 AM, Tony Stocker wrote:
> James,
>
> Okay, we've tested this and everything works fine now. I do have two
> quick questions:
>
> 1. Can you describe what we're doing "wrong" with our metadata?
> I mean we're obviously doing something that you're not expecting
> if you're having to maintain a code block with a comment with our
> name in it. I can't promise anything, but I'd at least like to
> pass this along to the scientists and algorithm developers to see
> if perhaps they could agree on making changes to make the metadata
> more "friendly".
>
> 2. I tried to install the latest HDF5 handler while doing this
> 1.7.1 rpm install, but unfortunately the latest version of the
> HDF 5 rpms is now 1.8.7 and your handler wants 1.8.5 only. So my
> question is, is there any way to make your handler rpms accept an
> => (equal to or greater) with regards to the underlying library
> dependency version? Or do things absolutely have to be fixed to a
> particular rpm version?
>
> Thanks again for all the help!
>
> Tony
>
>
> On Tue, 6 Sep 2011, Gallagher James wrote:
>
>>
>> On Sep 6, 2011, at 6:13 AM, Tony Stocker wrote:
>>
>>> James,
>>>
>>> Is there an updated package that we can install and use to see if the
>>> original problem is fixed?
>>
>> Yes - The Hyrax 1.7.1 has a new package for the HDF4 handler that should
>> work as a drop in replacement. I did not test that for both the 32 and
>> 64-bit linux builds, but it's likely to work. In the worst case, the changes
>> outside of the hdf4 handler and the BES (but not a change to the BES's ABI).
>>
>> Here's a link to the release: http://www.opendap.org/download/hyrax/1.7.1
>>
>> Here's a link to the 64-bit Linux RPM for the hdf4 handler:
>> http://www.opendap.org/pub/binary/hdf4_handler/centos5.2/3.9.3_x86_64
>>
>> James
>>
>>>
>>> Tony
>>>
>>> On Wed, 17 Aug 2011, Gallagher James wrote:
>>>
>>>>
>>>> On Aug 17, 2011, at 2:41 PM, Gallagher James wrote:
>>>>
>>>>>
>>>>> On Aug 17, 2011, at 12:50 PM, Tony Stocker wrote:
>>>>>
>>>>>>
>>>>>> James,
>>>>>>
>>>>>> I've sent your write-up along to our algorithm developers. I will say
>>>>>> that this issue is present in all of our HDF files in version 7,
>>>>>> regardless of algorithm. Beyond that I'm afraid that as systems guy,
>>>>>> I'm not in much of position to comment on the approach.
>>>>>
>>>>> I'm looking at the hdf4 handler and there are number of issues there. One
>>>>> is that the handler seems to think that every dimension of every array
>>>>> must have a name, so those without a name in the data file get names
>>>>> 'fakeDim_n' where n is 0, 1, 2, ... I'd like to silence that and also
>>>>> remove the attributes that are also being created for those 'fake dims'
>>>>> because it looks like those are only the 'fake dim' names and nothing
>>>>> more. It's easy to use the API to read the names (or discover there's no
>>>>> name associated with a particular dimension) from libdap or any one of
>>>>> the other APIs.
>>>>
>>>> Gads, I just found this comment in the handler code: // Reduce the fakeDim
>>>> list. FakeDim is found to be used by TRMM team.
>>>>
>>>> Comments?
>>>>
>>>> PS. I have fixed the bug that prompted this email w/o resorting to
>>>> removing the fakeDims or their 'name' attributes. I'd still like to remove
>>>> the latter if possible (that is, remove the fakeDim attribute 'String name
>>>> "fakeDim0"') but will wait on what you say.
>>>>
>>>>>
>>>>> Do you (and by extension NASA, THG) see any problem with my making the
>>>>> handler behave a little differently so long as we can get back the old
>>>>> behavior (I'll use a run-time parameter to switch on the new behavior)?
>>>>> It may be that the new behavior is required to read some files. I'm not
>>>>> sure about that last part, I have a fair amount more work to do on this
>>>>> problem.
>>>>>
>>>>> James
>>>>>>
>>>>>> Tony
>>>>>>
>>>>>>> On Tue, Aug 16, 2011 at 6:31 PM, Gallagher James <address@hidden> wrote:
>>>>>>>>
>>>>>>>> On Aug 11, 2011, at 1:30 PM, Tony Stocker wrote:
>>>>>>>>
>>>>>>>>> James,
>>>>>>>>>
>>>>>>>>> Any word on this issue?
>>>>>>>>
>>>>>>>> Tony,
>>>>>>>>
>>>>>>>> Are the VData variables (I'm not sure if 'variables' is the right
>>>>>>>> word) empty? When I look at the two kinds of files, in the v7 file
>>>>>>>> there are a number of VData that, when viewed with vshow, look like:
>>>>>>>>
>>>>>>>> vs:3 <1962/10340> nv=0 i=0 fld [SDS variable] vsize=4 (NoName {SDSVar})
>>>>>>>>
>>>>>>>> That is the field is called 'SDS variable', the name is NoName and nv
>>>>>>>> is 0. In the v6 files (the ones that work) there are no entries with
>>>>>>>> nv=0; all of the nv values are >= 1 in the files that work.
>>>>>>>>
>>>>>>>> In the documentation for VSsetfield() (which must be called before a
>>>>>>>> Vread()) it says that if the VData is empty (I'm paraphrasing here)
>>>>>>>> then the function will return -1 (the code for failure). However,
>>>>>>>> prior to calling that function we are testing a number of other
>>>>>>>> things. These VData have all of the following: a fieldname,
>>>>>>>> fieldorder, fieldsize, and fieldtype.
>>>>>>>>
>>>>>>>> I will try building the handler so that it treats this particular
>>>>>>>> error as meaning the vdata is empty, but it's very hard for me to tell
>>>>>>>> what's what in these very large HDF4 files. While I work on that, can
>>>>>>>> you let me know if this seems to be a promising tack?
>>>>>>>>
>>>>>>>> HDF folks: Can you weigh in on this? Am I close to understanding the
>>>>>>>> semantics of VSsetfield()?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> James
>>>>>>>>
>>>>>>>> PS. The ticket is http://scm.opendap.org:8090/trac/ticket/1793
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Tony
>>>>>>>>>
>>>>>>>>> On Thu, 21 Jul 2011, address@hidden wrote:
>>>>>>>>>
>>>>>>>>>> Tony,
>>>>>>>>>> I'm out for the next few days, but do send those files. The problem
>>>>>>>>>> is in the hdf4 handler. Most likely anyway.
>>>>>>>>>> James
>>>>>>>>>> Sent from my Verizon Wireless Phone
>>>>>>>>>> ----- Reply message -----
>>>>>>>>>> From: "Tony Stocker" <address@hidden>
>>>>>>>>>> Date: Thu, Jul 21, 2011 2:40 pm
>>>>>>>>>> Subject: [support] Re: Could not read vdata records
>>>>>>>>>> To: "OPENDAP support" <address@hidden>
>>>>>>>>>> Cc: "Owen Kelley" <address@hidden>
>>>>>>>>>> And now to actually include the attachments (*sigh*)
>>>>>>>>>> On Thu, 21 Jul 2011, Tony Stocker wrote:
>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>
>>>>>>>>>>> Okay, quick and dirty description of the problem. We have started
>>>>>>>>>>> reprocessing our science data, i.e. developers have new algorithms
>>>>>>>>>>> and so we
>>>>>>>>>>> re-run everything. Our old data is V6 and the new data is V7. We
>>>>>>>>>>> keep both
>>>>>>>>>>> around for comparison purposes.
>>>>>>>>>>>
>>>>>>>>>>> OpenDAP has been working fine with our V6 data, and continues to do
>>>>>>>>>>> so.
>>>>>>>>>>>
>>>>>>>>>>> However it is choking on all of our V7 data, producing the error
>>>>>>>>>>> message:
>>>>>>>>>>> "libdap exception building response: error_code = 1001: Could not
>>>>>>>>>>> read Vdata
>>>>>>>>>>> records."
>>>>>>>>>>>
>>>>>>>>>>> I'm going attaching two text files containing debug output of a
>>>>>>>>>>> successful V6
>>>>>>>>>>> open and an unsuccessful V7 open of the same day/orbit:
>>>>>>>>>>> od.v6.19980101.1c21.00539-debug.txt i.e. WORKING FILE
>>>>>>>>>>> od.v7.19980101.1c21.00539-debug.txt i.e. NOT working FILE
>>>>>>>>>>>
>>>>>>>>>>> I also wanted to see if you guys would let me send you both of
>>>>>>>>>>> these HDF
>>>>>>>>>>> files (the v6 and v7 ones) to see if you guys see the same behavior
>>>>>>>>>>> on your
>>>>>>>>>>> installations. I haven't attached these to this email however
>>>>>>>>>>> since each is
>>>>>>>>>>> ~150 MB in size. I think it would be helpful to see if the problem
>>>>>>>>>>> is
>>>>>>>>>>> universal or localized to our installation. Let me know if you're
>>>>>>>>>>> willing to
>>>>>>>>>>> take and test the files, and how you'd like to get them (email,
>>>>>>>>>>> ftp, etc.)
>>>>>>>>>>>
>>>>>>>>>>> I also want to mention that we have verified that the V7 HDF files
>>>>>>>>>>> are not
>>>>>>>>>>> corrupt or otherwise unreadable in themselves by using two other HDF
>>>>>>>>>>> viewer/reader programs on them, and they come up fine, our only
>>>>>>>>>>> problem seems
>>>>>>>>>>> to be with the OpenDAP server reading them.
>>>>>>>>>>>
>>>>>>>>>>> Here are our software product versions:
>>>>>>>>>>> bes-3.9.0-1
>>>>>>>>>>> bes-devel-3.9.0-1
>>>>>>>>>>> dap-server-4.1.0-1
>>>>>>>>>>> libdap-3.11.0-1
>>>>>>>>>>> libdap-devel-3.11.0-1
>>>>>>>>>>> olfs-1.7.1
>>>>>>>>>>> freeform_handler-3.8.2-1
>>>>>>>>>>> hdf4_handler-3.9.1-1
>>>>>>>>>>> hdf5_handler-1.4.3-1
>>>>>>>>>>> netcdf_handler-3.9.2-1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Look forward to hearing back from you guys, since this is
>>>>>>>>>>> perplexing and
>>>>>>>>>>> pretty much out of our balliwick.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Tony
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> +-----------------------------------+
>>>>>>>>>> | Tony Stocker |
>>>>>>>>>> | Systems Programmer |
>>>>>>>>>> | TSDIS/TRMM Code 610.2 |
>>>>>>>>>> | 301-614-5738 (office) |
>>>>>>>>>> | 301-614-5269 (fax) |
>>>>>>>>>> | address@hidden |
>>>>>>>>>> +-----------------------------------+
>>>>>>>>>> --
>>>>>>>>>> This message has been scanned for viruses and
>>>>>>>>>> dangerous content by MailScanner, and is
>>>>>>>>>> believed to be clean.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> +-----------------------------------+
>>>>>>>>> | Tony Stocker |
>>>>>>>>> | Systems Programmer |
>>>>>>>>> | TSDIS/TRMM Code 610.2 |
>>>>>>>>> | 301-614-5738 (office) |
>>>>>>>>> | 301-614-5269 (fax) |
>>>>>>>>> | address@hidden |
>>>>>>>>> +-----------------------------------+
>>>>>>>>> --
>>>>>>>>> This message has been scanned for viruses and
>>>>>>>>> dangerous content by MailScanner, and is
>>>>>>>>> believed to be clean.
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> James Gallagher
>>>>>>>> jgallagher at opendap.org
>>>>>>>> 406.723.8663
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> +-----------------------------------+
>>>>>> | Tony Stocker |
>>>>>> | Systems Programmer |
>>>>>> | TSDIS/TRMM Code 610.2 |
>>>>>> | 301-614-5738 (office) |
>>>>>> | 301-614-5269 (fax) |
>>>>>> | address@hidden |
>>>>>> +-----------------------------------+
>>>>>> --
>>>>>> This message has been scanned for viruses and
>>>>>> dangerous content by MailScanner, and is
>>>>>> believed to be clean.
>>>>>>
>>>>>
>>>>> --
>>>>> James Gallagher
>>>>> jgallagher at opendap.org
>>>>> 406.723.8663
>>>>>
>>>>
>>>> --
>>>> James Gallagher
>>>> jgallagher at opendap.org
>>>> 406.723.8663
>>>>
>>>>
>>>
>>> --
>>> +-----------------------------------+
>>> | Tony Stocker |
>>> | Systems Programmer |
>>> | TSDIS/TRMM Code 610.2 |
>>> | 301-614-5738 (office) |
>>> | 301-614-5269 (fax) |
>>> | address@hidden |
>>> +-----------------------------------+
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>> --
>> James Gallagher
>> jgallagher at opendap.org
>> 406.723.8663
>>
>>
>
> --
> +-----------------------------------+
> | Tony Stocker |
> | Systems Programmer |
> | TSDIS/TRMM Code 610.2 |
> | 301-614-5738 (office) |
> | 301-614-5269 (fax) |
> | address@hidden |
> +-----------------------------------+
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
--
James Gallagher
jgallagher at opendap.org
406.723.8663