Chi,
Connecting via I1 (TLS) would need to be coordinated with the NOC.
Cameron
Chi.Y.Kang wrote:
Justin, is all the data feeds getting into LDM2 and LDM1 ? Also, is
there a way we can test the connection to the LDM system via over I1
rather then I2?
Douglas Schuster wrote:
Hi Justin,
The latencies are still looking bad using ldm2. This has lead to the
continuation of large numbers
of missing fields (identical on both receiving machines, NCAR and
Unidata) from all model cycles over the weekend,
continuing this morning.
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.unidata.ucar.edu+LOG
Doug
On Jun 18, 2007, at 6:32 AM, Justin Cooke wrote:
You're right Chi, I misread the graph, we are on ldm2 and have been
since Saturday. My apologies.
Justin
Chi Y Kang wrote:
Justin Cooke wrote:
Chi,
It looks like you switched us back to ldm1 on Saturday but
according to the graphs from Steve they experienced the same delays.
Running on ldm2 right now. It looks like the send-Q on our end seems
to be okay. All of these connections are going via I2, let me see
what rate limit there is set on I2 connection coming out of the campus.
Justin
Steve Chiswell wrote:
Chi & Justin,
The latency of data today has been high like yesterday, even with
the switch of
ldm2. The throughput looks restricted either by a router or
firewall/packet
shaping, but was wondering if coincident with Justin's restart was
that the
connections had to be re-established, so changes took effect at
that time.
Thanks for all your efforts,
Steve Chiswell
Unidata User Support
On Fri, 15 Jun 2007, Chi Y Kang wrote:
Wait a minute here,
128.117.140.208 isn't in the mix. The other hosts are.
I updated the LDM access list. Should we just have some class C
ranges
to have access rather then ip at a time?
Also, i noticed that the send-Q are pretty normal on ldm2 server
right
now but was pretty high on ldm1. might be just an issue with the ACL
list.
128.117.12.2
128.117.12.3
128.117.130.220
128.117.140.208
128.117.140.220
128.117.149.220
128.117.156.220
128.174.80.16
128.174.80.47
140.90.193.19
140.90.193.227
140.90.193.228
140.90.193.99
140.90.226.201
140.90.226.202
140.90.226.203
140.90.226.204
140.90.37.12
140.90.37.13
140.90.37.15
140.90.37.16
140.90.37.40
144.92.130.88
144.92.131.244
150.9.117.128
192.12.209.57
192.58.3.194
192.58.3.195
192.58.3.196
192.58.3.197
193.61.196.74
198.181.231.53
208.64.117.128
Justin Cooke wrote:
Chi,
The reboot doesn't seem to have helped. Is there anything else
that may
be causing these issues? Network related after I performed the
restart
of LDM? Steve has a few possibilities:
/It seems to be network related at your end, but strange that it
occurred at the time when you retsrtaed the LDM- unless there
was some
sort of firewall or packet filter that occurred when the LDM's
re-connected. /
Justin
Steve Chiswell wrote:
Justin,
I haven't seen any improvement from ncepldm to the top level
relays
daffy.unidata.ucar.edu (Unidata), idd.aos.wisc.edu (U. WIsconsin),
flood.atmos.uiuc.edu (U. Illinois) or atm.cise-nsf.gov (NSF, DC).
It seems to be network related at your end, but strange that it
occurred
at the time when you retsrtaed the LDM- unless there was some
sort of
firewall or packet filter that occurred when the LDM's
re-connected.
Thanks for your time in looking at this,
Steve
On Fri, 2007-06-15 at 15:31 -0400, Justin Cooke wrote:
Steve and Doug,
I just got a call from Chi at the WOC, he rebooted LDM1 after
noticing
an unusual load on the machine. LDM is again running on that
box and it
remains primary, can you check to see how the latencies are now?
Thanks,
Justin
Doug Schuster wrote:
Justin,
28,079 products are missing from the 12z cycle. You'll be
getting the
automated email shortly.
-Doug
On Jun 15, 2007, at 12:48 PM, Justin Cooke wrote:
Steve,
I've turned off the feed to LDM2.
There is no other load on the ldm1 system except for LDM.
Doug, are you missing many of the TIGGE params for 12Z?
Justin
Steve Chiswell wrote:
Justin,
That didn't change the behavior. Still seeing latency.
perhaps turning off the other feed. Is there any load
other than LDM on the system?
Steve
On Fri, 2007-06-15 at 12:56 -0400, Justin Cooke wrote:
Steve,
I've recreated the queue, let me know if you are still
seeing issues.
If so I'll turn off the feed to ldm2 to see if that
corrects things.
Justin
Steve Chiswell wrote:
Justin,
I don't know if they saw a disk space problem with
log files not being rotated, but it might just be
best today to build a new queue:
ldmadmin stop
ldmadmin delqueue
ldmadmin mkqueue
ldmadmin start
That will mean some queued data would be lost, but if
users aren't
getting it
anyway, then its best to ensure that the queue isn't
corrupt for the
weekend.
Happy Friday....
Thanks,
Steve
On Fri, 2007-06-15 at 12:13 -0400, Justin Cooke wrote:
Steve,
Our logs on the primary ldm system "ldm1" had not
rotated for
nearly a week. I sent email to the WOC support and this
was the
response:
Looks like the seed file was missing after we brought
the system
backup
from the last outage. should be good now.
Justin Cooke wrote:
WOC,
I noticed that our logs for LDM have not been rotated
on machine
ldm1
since 06/05/2007. We have a cron entry that runs "ldmadmin
newlog" at
00Z every day.
I attempted to run the command by hand and got the
following back:
ldm@ldm1:~$ bin/ldmadmin newlog
hupsyslog: couldn't open /var/run/syslogd.pid
I checked but /var/run/syslogd.pid is not there but it
is on ldm2.
Could there be a problem with syslogd on ldm1?
Justin
Also around that time I turned on our backup feed to the
ldm2
system which had been off since that system had issues a
few
weeks ago (we were asked by WOC to turn it back on). I
have sent
email to their support group asking if both ldm1 and
ldm2 are
responding to the ncepldm.woc.noaa.gov address or if
something
else is going on.
Justin
Steve Chiswell wrote:
Justin,
Yesterday just after 18Z, the data flow from
ncepldm.woc.noaa.gov
to top level sites at NSF and Unidata both began
showing high
latency:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT
+atm.cise-nsf.gov
and
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT
+daffy.unidata.ucar.edu
Data volume out has dropped as a result:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT
+atm.cise-nsf.gov
Since the behavior is similar at both sites at separate
locations, the
problem would appear to be near your end. Since that
coincides
with your
restart of the LDM, could you fill me in on the issues
you were
experiencing?
Thanks
Steve Chiswell
Unidata User Support
On Fri, 2007-06-15 at 11:38 -0400, Justin Cooke wrote:
Doug,
I had to restart our LDM yesterday right before the
18Z cycle,
we had an issue with out logging but none of the
configuration
files changed. Could one of your feeds have lost the
connection
to our LDM during that restart?
Justin
Douglas Schuster wrote:
Yes, we've received partial cycles. More than half of
the
expected fields have been missing
in each cycle from June 14 18Z, to June 15, 06Z. The
number
of missing fields varies between
each cycle.
Doug
On Jun 15, 2007, at 9:11 AM, Justin Cooke wrote:
Doug,
Have you received any GEFS data from us today? Or is
it just
certain fields you are missing?
Justin
--Chi Y. Kang
Contractor
Principal Engineer
Phone: 301-713-3333 x201
Cell: 240-338-1059
|