[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Using HttpClient/HttpClientManager in NJ 4.2

Hi Robert:

1. ok, we see the multiple challenge problem and are working on a solution.
2. slow download is mostly due to latency, because of the way HTTPRandomAccessFile works. Each "seek" causes a round-trip (!). Theres no fix for this, better is to use an opendap server for performance.
3. We can probably cut round-trips in half by doing pre-emptive authorization. Im not sure if that used to work in 4.2.18 or not. We will try to get that working also. 

4. We will fix the "user agent" problem in the next release also.
Thanks for reporting the problem, we will get back to you when we have a fix. 

John


On 3/11/2011 12:49 PM, Robert B. Schmunk wrote:


John,

The test files are at

http://www.giss.nasa.gov/staff/rschmunk/test/file1.nc
http://www.giss.nasa.gov/staff/rschmunk/test/file2.nc

Try userid 'jcaron' and password 'boulder'.

Another problem that I encounter with authentication
in 4.2.21 is that after I have successfully loaded a
file from a remote directory (albeit having to enter
the password twice), when I request a second file from
the same directory, it requests authentication again.
This is not the case in 4.2.18.

The problem that I reported regarding the slow download
is apparently not caused by the authentication issues,
although it's possible that it is exacerbated by it.
The csae in which I am primarily running into occurs
when I create a lat-vert plot from an array with lon,
lat and vert dimensions. I'm averaging over all the
points in the lon dimension, but am getting each slice
along that dimension individually, which means a lot of
individual requests being fired off to the remote server.

rbs






On Mar 11, 2011, at 09:12, John Caron wrote:


Hi Robert:

We have been refactoring that code, so im guessing we havent got it right yet.

Can I get access to the remote netcdf file so I can recreate?

John

On 3/8/2011 9:58 PM, Robert B. Schmunk wrote:

John,

I'm trying to diagnose some problems using the NJ libraries to
access a remote netCDF dataset located in a website directory
that requires basic HTTP authentication. It's getting a bit
frustrating, so I'm writing for any advice you might have.

I've created a CredentialsProvider dialog to handle the necessary
user input, and in general, it seems to work okay except that...

The first problem is that using NJ 4.2.21, when one enters an ID
and password and the result, the first attempt to enter these
credentials is _always_ rejected, and one is asked to enter them
again. The second time, it always works (assuming the ID and
password are correct, of course).

Once access is gained, I found that the data downloads were
taking much too long. A check of the webserver logs revealed
that the incoming requests were alternating between no
credentials received and a 401 response code, and receiving
the credentials but only passing back 20 kB of data.

I reverted to using NJ 4.2.18, which I had been using for a
couple months. The first problem above went away. The second
remained.

As a side issue, I found that with 4.2.21, the user agent info
received by the server was "Jakarta Commons-HttpClient/3.1",
but with 4.2.18, it included the user agent info that I
passed when creating the HttpClient object.

My code when using NJ 4.2.21 looks like

     public static void initHttpClient (
         CredentialsProvider cprov)
     {
         HttpClientManager.init (cprov,
             "Panoply/3.0 (http://www.giss.nasa.gov/tools/panoply/); ");
     }

My code when using 4.2.18 looks like

     public static void initHttpClient (
         CredentialsProvider cprov)
     {
         HttpClient client = HttpClientManager.init (cprov,
             "Panoply/3.0 (http://www.giss.nasa.gov/tools/panoply/); ");
         HTTPRandomAccessFile.setHttpClient (client);
         CdmRemote.setHttpClient (client);
         NetcdfDataset.setHttpClient (client);
         DConnect2.setHttpClient (client);
     }

Thanks,

rbs


--
Robert B. Schmunk, address@hidden
NASA Goddard Institute for Space Studies, 2880 Broadway, New York, NY 10025





--
Robert B. Schmunk, address@hidden
NASA Goddard Institute for Space Studies, 2880 Broadway, New York, NY 10025