wow, i didnt envision this scale. It should work if you dont run out of memory, but probably need to think of better algorithms.
however, once its indexed, shouldnt do it again, if you use
<update startup="nocheck" trigger="allow" />
On 5/28/2013 9:17 AM, Dan Swank - NOAA Affiliate wrote:
Morning John,
Assume your still waiting on logs.. will get thta
to you shortly. Looks like I stepped into
a 14 day process with indexing the NDFD
4667691 gbx9's created over the weekend and its
still got about 15 million to go. The spin-up I
mentioned took approximate 15 hours. So unless
I comment the NDFD away, restarting will cost alot of time.
On Thu, May 23, 2013 at 4:06 PM, Dan Swank - NOAA Affiliate<address@hidden <mailto:address@hidden>> wrote:
One quick note
the ruc130anl FC is actually working from
the openDap form. ruc252anl is not. the
errors still exist in the ruc130anl logs .. however.
But it does not seem to be affecting functionality.
On Thu, May 23, 2013 at 3:37 PM, Dan Swank - NOAA Affiliate<address@hidden <mailto:address@hidden>> wrote:
John,
After opening it up to the entire NDFD tree.... the scan log
seems to
cease here.... wondering if there is a "spin up" period while its
finding files ... and nothing is written to the log for a period
of time ?
(talking scale of millions of files )
Probably just need to give it more time.
2013-05-23T15:12:35.655 -0400 INFO - FeatureCollection added =
name ='NDFD' type='GRIB'
spec='/global/nomads/nexus/ndfd/**/......_...._#yyyyMMdd#....$'
timePartition =1 day
updateConfig =UpdateConfig{startup=true, recheckAfter='null',
rescan='null', triggerOk=true, deleteAfter=null}
protoConfig =ProtoConfig{choice=Penultimate, change='null',
param='null', outerNcml='null', cacheAll=true}
hasInnerNcml =false
gribConfig =GribConfig: datasetTypes=[Best, Files]
NNR2 :
After battling a bit with the spec and mark, I got this:
-- seemingly odd exception.
---------
2013-05-23T15:23:55.682 -0400 INFO - FeatureCollection added =
name ='NNR2pgrbft' type='GRIB'
spec='/global/nomads/nexus/reanalysis-2/6hr/pgb.ft06/pgb.ft06.#yyyyMM'
timePartition =1 year
updateConfig =UpdateConfig{startup=true, recheckAfter='null',
rescan='null', triggerOk=true, deleteAfter=null}
protoConfig =ProtoConfig{choice=Penultimate, change='null',
param='null', outerNcml='null', cacheAll=true}
hasInnerNcml =false
gribConfig =GribConfig: datasetTypes=[Best, Files]
2013-05-23T15:27:21.062 -0400 INFO - NNR2pgrbft : was scanned
nfiles= 360
2013-05-23T15:27:21.062 -0400 INFO - NNR2pgrbft: createIndex
/home/tomcat/tomcat-main.4.3.15/content/thredds/cache/GribIndex/global/nomads/nexus/reanalysis-2/6hr/pgb.ft06/NNR2pgrbft.ncx
2013-05-23T15:27:21.063 -0400 ERROR - Fail to create/update
collection
java.lang.RuntimeException: SimpleDateFormat = yyyyMM fails on
pgb.ft06.198306 ParseException:Unparseable date: "pgb.ft06.198306"
at
ucar.nc2.units.DateFromString.getDateUsingCompleteDateFormatWithOffset(DateFromString.java:214)
at
ucar.nc2.units.DateFromString.getDateUsingDemarkatedCount(DateFromString.java:99)
at
thredds.inventory.DateExtractorFromName.getDate(DateExtractorFromName.java:67)
at
thredds.inventory.DateExtractorFromName.getCalendarDate(DateExtractorFromName.java:74)
~~~~
---
This is probably all from me today,
See you Tuesday.
On Thu, May 23, 2013 at 8:56 AM, Dan Swank - NOAA Affiliate<address@hidden <mailto:address@hidden>> wrote:
Okay, I will flush the .ncx for ruc.
NDFD is purring now for 1 days worth.
It will take a long time to generate gbx9 for the entire
online period,
we will see how far it gets over the extended weekend.
That will be a great stress test.
In the meantime,
I will take a shot at the GDS hashes for NDGD.
On Wed, May 22, 2013 at 9:04 PM, John Caron<address@hidden <mailto:address@hidden>> wrote:http://nomads.ncdc.noaa.gov/__data/gfsanl/201001-test/__tdsLogs-20130522.tar.bz2
havent been able to reproduce this one.
delete all the collection ncx and restart tds. then send
me the logs.
im gone starting tommorrow for a long weekend, so we
will resume next tuesday
ciao!
On 5/22/2013 11:57 AM, Dan Swank - NOAA Affiliate wrote:
Got the correct version running this morning.
Heres some logs, some insight on the fc.ruc252,
but not too much<mailto:address@hidden
<http://nomads.ncdc.noaa.gov/data/gfsanl/201001-test/tdsLogs-20130522.tar.bz2>
On Tue, May 21, 2013 at 5:38 PM, John Caron
<address@hidden <mailto:address@hidden>==============================__=========
<mailto:address@hidden>__>> wrote:
On 5/21/2013 2:40 PM, Dan Swank - NOAA
Affiliate wrote:
Running Version 4.3.17-SNAPSHOT - 20130515.1958
then it should say:
Version 4.3.17-SNAPSHOT - 20130517.1111
--<mailto:address@hidden <mailto:address@hidden>>
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: address@hidden
<mailto:address@hidden>
Phone: 828-271-4007 <tel:828-271-4007>
==============================__============================================__==============
Any opinions expressed in this message are mine
personally and
do not represent any position from STG, ERT, NCDC,
or NOAAPhone: 828-271-4007 <tel:828-271-4007>
--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: address@hidden <mailto:address@hidden>Email: address@hidden <mailto:address@hidden>
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================
--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007 <tel:828-271-4007>Email: address@hidden <mailto:address@hidden>
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================
--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Phone: 828-271-4007 <tel:828-271-4007>
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================
--
=======================================
Dan Swank
NCDC-NOMADS Project
Team ERT/STG
(NCDC Government Contractor)
Data Access & Applications Branch
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Email: address@hidden <mailto:address@hidden>
Phone: 828-271-4007
============================================
Any opinions expressed in this message are mine personally and
do not represent any position from STG, ERT, NCDC, or NOAA
============================================