Re: [Fwd: Re: problems reading GFS data from NOMADS-HD; DConnect]

Hi John,

On Fr, 23.04.2010, 17:02, John Caron wrote:
> Hi Dan, Glenn:
>
> java.net.SocketException: Connection reset
>
> probably means that the socket was closed on the server.
>
> To go further with this we need to look at the threddsServlet.log files
> and see if we are getting any error messages in there that correspond to
> what Gerald is seeing. this may be hard without knowing the exact URL
> and a small time window during which the errors happened, and maybe the
> client IP address. Gerald do you have any of that?
I was testing the downloading from NOAMDS HD today, it worked most of the
time, but one time in the morning and one time just now (around 17:15 UTC)
I received again the same error message. The last error happened when when
reading the GFS forecast (variable tmpprs) from 2010-04-16, 00UTC, from
file
dods://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100416/gfs_hd_00z.
Since our system administrator is not here any longer today, I don't know
the client IP. Sorry, also no network guys around.
If the connection gets very slow, shouldn't there just be a "connection
timed out" error message?
Regards, Gerald

>
> this can also happen when the connection gets very slow, eg if one of
> the IP routes is flaky. Gerald can you ask your network guys (if you
> have them) to test to see if your network routing to nomads has any
> problems?


>
> On 4/23/2010 6:59 AM, Dan Swank wrote:
>> John,
>> This addy is for NCDC-NOMADS servers nomads.ncdc.* only.
>> For NCEP-NOMADS ~ you should contact their HelpDesk: @
>> http://nomads.ncep.noaa.gov/txt_descriptions/Help_Desk_doc.shtml
>>
>> -Dan
>>
>>
>> address@hidden wrote the following on 4/22/2010 8:41 PM:
>>
>>> hmm-
>>> John
>>> can you assist here?
>>>
>>> thx, glenn
>>>
>>> ----- Original Message -----
>>> From: Gerald Spreitzhofer<address@hidden>
>>> Date: Thursday, April 22, 2010 4:28 pm
>>> Subject: Re: [Fwd: Re: problems reading GFS data from NOMADS-HD;
>>> DConnect]
>>>
>>>
>>>> Thanks, John.
>>>> Glenn, could you please answer my mail or explain me the error
>>>> messages I
>>>> am receiving from time to time (see below)? I think there were similar
>>>> messages some days in fall, some people from the mailing list
>>>> complainedand then you were able to fix them.
>>>> Best regards, Gerald
>>>>
>>>> On Do, 22.04.2010, 22:18, John Caron wrote:
>>>>
>>>>> gerald, send reports to
>>>>>
>>>>> address@hidden
>>>>>
>>>>>
>>>>>
>>>>> On 4/22/2010 12:54 PM, Gerald Spreitzhofer wrote:
>>>>>
>>>>>> John,
>>>>>> I am having the same problems as before and the same error
>>>>>>
>>>> message (see
>>>>
>>>>>> below, java.net.SocketException) most of the day, using
>>>>>>
>>>> DODS/OPENDAP.>>  Also
>>>>
>>>>>> just clicked again on the motherlode-link, there I can download GFS
>>>>>> 0.5deg
>>>>>> data (well, some gridpoints, I did not check for big amounts of
>>>>>>
>>>> data).>>  The
>>>>
>>>>>> amount of data I am downloading is max. 20 MB (like always), I
>>>>>>
>>>> don't>>  think
>>>>
>>>>>> this is a problem. The error usually ocurs some time during the
>>>>>> downloading, not just at the beginning.
>>>>>> Can you forward this mail to persons who might help?
>>>>>> Thanks, Gerald
>>>>>>
>>>>>> On Mo, 19.04.2010, 16:44, john caron wrote:
>>>>>>
>>>>>>
>>>>>>> gerald, can you try hitting datasets on
>>>>>>> http://motherlode.ucar.edu/thredds/ and see if you have any
>>>>>>>
>>>> problems?>>>
>>>>
>>>>>>> Gerald Spreitzhofer wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>> Thanks for answering my mail last Friday that quickly.
>>>>>>>>
>>>> Problems still
>>>>
>>>>>>>> persist, but it looks like they are not permanent (sometimes in
>>>>>>>> between
>>>>>>>> GFS data can be read without problems). Maybe the error
>>>>>>>>
>>>> messages below
>>>>
>>>>>>>> help (they were produced this morning around 8 UTC).
>>>>>>>> Best Regards, Gerald
>>>>>>>>
>>>>>>>> ===== Reading 3d-variable 58: pressfc=====
>>>>>>>> java.net.SocketException: Connection reset
>>>>>>>>        at java.net.SocketInputStream.read(SocketInputStream.java:168)
>>>>>>>>        at
>>>>>>>>
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:183)>>>>
>>>>    at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
>>>>
>>>>>>>>        at
>>>>>>>>
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:277)>>>>
>>>>    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:741)
>>>>
>>>>>>>>        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:702)
>>>>>>>>        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:712)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:583)>>>>
>>>     at dods.dap.DConnect.openConnection(DConnect.java:174)
>>>
>>>>>>>>        at dods.dap.DConnect.getDataFromUrl(DConnect.java:380)
>>>>>>>>        at dods.dap.DConnect.getData(DConnect.java:339)
>>>>>>>>        at dods.dap.DConnect.getData(DConnect.java:522)
>>>>>>>>        at
>>>>>>>>
>>> ucar.nc2.dods.DODSNetcdfFile.readDODSData(DODSNetcdfFile.java:536)>>>>
>>> at ucar.nc2.dods.DODSVariable.readDODSData(DODSVariable.java:497)
>>>
>>>>>>>>        at ucar.nc2.dods.DODSVariable.read(DODSVariable.java:393)
>>>>>>>>        at SEVIS.NetcdfUtil.readNcArrayLayer(NetcdfUtil.java:1846)
>>>>>>>>        at SEVIS.NetcdfUtil.readNcArray(NetcdfUtil.java:1708)
>>>>>>>>        at
>>>>>>>>
>>>> SEVIS.NetcdfUtil.readNcArray_0MedAdapted(NetcdfUtil.java:1659)>>>>
>>>>    at SEVIS.DODSReader.writeNcVariables(DODSReader.java:825)
>>>>
>>>>>>>>        at SEVIS.DODSReader.createNetcdfFile(DODSReader.java:676)
>>>>>>>>        at SEVIS.DODSReader.<init>(DODSReader.java:159)
>>>>>>>>        at SEVIS.SuperDataFile.getDodsFile(SuperDataFile.java:2526)
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>> SEVIS.SuperDataFile.getSpecifiedNetcdfFile(SuperDataFile.java:2344)>>>>
>>>     at
>>> SEVIS.MainFrame.forecastUpdate(MainFrame.java:2131)
>>>
>>>>>>>>        at
>>>>>>>>
>>>>>>>>
>>>> SEVIS.MainFrame.jButtonDataUpdate_actionPerformed(MainFrame.java:2163)>>>>
>>>>
>>>>>>>> ------------------------ Ursprüngliche Nachricht
>>>>>>>> ------------------------
>>>>>>>> Betreff: Re: problems reading GFS data from NOMADS-HD; DConnect
>>>>>>>> Von:     "Glenn Rutledge"<address@hidden>
>>>>>>>> Datum:   Fr, 16.04.2010, 19:41
>>>>>>>> An:      "John Caron"<address@hidden>
>>>>>>>> Cc:      "Gerald Spreitzhofer"<address@hidden>
>>>>>>>>            address@hidden
>>>>>>>> ---------------------------------------------------------------
>>>>>>>>
>>>> -----------
>>>>
>>>>>>>> Thanks John,
>>>>>>>> We don't seem to be experiencing any issues.  I've cc:ed Dan
>>>>>>>>
>>>> Swank>>>>  here
>>>>
>>>>>>>> and he might be able to track down this issue.
>>>>>>>> Regards, Glenn
>>>>>>>>
>>>>>>>> John Caron said the following on 4/16/2010 1:01 PM:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi gerald: im not having any trouble with that URL. is it still
>>>>>>>>> happening (11 am MST) ?
>>>>>>>>>
>>>>>>>>> On 4/16/2010 10:15 AM, Gerald Spreitzhofer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>> I am operationally reading GFS 0.5deg data from the NOMADS-
>>>>>>>>>>
>>>> HD server
>>>>
>>>>>>>>>> via
>>>>>>>>>> DODS and the Java/NetCDF library. I did not change anything
>>>>>>>>>>
>>>> in my
>>>>
>>>>>>>>>> code
>>>>>>>>>> during the last couple of months, but recently (for instance
>>>>>>>>>>
>>>> the>>>>>>  last
>>>>
>>>>>>>>>> 20
>>>>>>>>>> hours) there have been some errors reading data from there
>>>>>>>>>>
>>>> (see the
>>>>
>>>>>>>>>> protocol below). I think I remember that there was a similar
>>>>>>>>>>
>>>> problem>>>>>>  last
>>>>
>>>>>>>>>> fall, and the reason was a change in some security or firewall
>>>>>>>>>> setting on
>>>>>>>>>> NOMADS, just concerning this protocol. Do you know what the
>>>>>>>>>>
>>>> reason>>>>>>  for the
>>>>
>>>>>>>>>> messages can be or who might be responsible? Thanks, Gerald
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ===== Reading 3d-variable 28: hgtsfc=====
>>>>>>>>>> DConnect getData failed; retry (1,100)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (2,200)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (3,400)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (4,800)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (5,1600)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[5:5][250:299][704:719]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (1,100)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (2,200)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (3,400)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (4,800)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>> DConnect getData failed; retry (5,1600)
>>>>>>>>>>
>>>>>>>>>>
>>> http://nomads.ncep.noaa.gov:9090/dods/gfs_hd/gfs_hd20100415/gfs_hd_18z.dods?hgtsfc[0:0][250:299][0:66]>>>>>>
>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>> Dr. Gerald Spreitzhofer, Researcher
>>>>>>>>>>>> Institute of Meteorology and Geophysics
>>>>>>>>>>>> University of Vienna
>>>>>>>>>>>> A-1090 Vienna, Althanstrasse 14, UZA2
>>>>>>>>>>>> Tel.: +43-1-4277-537 33
>>>>>>>>>>>> email: address@hidden
>>>>>>>>>>>> -----------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>
>
>