[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [## 1441 ##] SNOTEL site TUGA2 temperature data issue Jan 03

Alex,

Thank you for all the information.
I reviewed your email and the attachment.

I think I understand what is going on.  The SNOTEL are remote sites and use quasi-reliable comms (like meteor-burst).
The comms can cause messages to be buffered up for several hours before being sent.

I looked in depth at TUGA2 for last night.  TUGA2 NRCS's code is 954:AK:SNTL and the UTC offset is 9 hours.

I think there are 2 main issues going on and I'm not sure how to best handle them.
- MADIS real-time processing only does the current and prior hour of data. 
- SYNOPTIC processing may not handle multiple old hours in one hour.

MADIS archive processing will pick up the old data, but it is available a day or so later and only in the netcdf file on the web.

I'll contact NRCS to verify they don't have something else going on, but this is not uncommon in the past.

First file the time periods are in the data from NRCS are below
loc 15  UTC 00: in file Jan  5 00:21 LDAD.receipt.snotel.obs.srt.2400500
loc 16  UTC 01: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 17  UTC 02: in file Jan  5 00:21 LDAD.receipt.snotel.obs.srt.2400500
loc 18  UTC 03: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 19  UTC 04: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 20  UTC 05: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 21  UTC 06: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 22  UTC 07: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 23  UTC 08: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 24  UTC 09: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 01  UTC 10: in file Jan  5 10:28 LDAD.receipt.snotel.obs.srt.2400510
loc 02  UTC 11: in file Jan  5 11:22 LDAD.receipt.snotel.obs.srt.2400511

Here are the first instances of each hour in the MADIS web SHEF file
loc 15  UTC 00: in file Jan  5 00:23 aksnotel.hourly.txt.20240105_0045
loc 16  UTC 01: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 17  UTC 02: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 18  UTC 03: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 19  UTC 04: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 20  UTC 05: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 21  UTC 06: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045 
loc 22  UTC 07: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 23  UTC 08: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 24  UTC 09: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 01  UTC 10: in file Jan  5 10:31 aksnotel.hourly.txt.20240105_1045
loc 02  UTC 11: in file Jan  5 11:25 aksnotel.hourly.txt.20240105_1145

-- Leon

On Thu, Jan 4, 2024 at 11:52 AM Leon Benjamin - NOAA Affiliate <address@hidden> wrote:
Alex,
Thanks. Let me check the email you sent.  I hope these are the same issues.

I'll start pulling the SHEF files and archiving them to verify the issue.

The data is in the logs, but there is a time check before the SHEF is written.

-- Leon

On Thu, Jan 4, 2024 at 11:33 AM Synoptic Support <address@hidden> wrote:
Hello Leon and All,

Following up from the Synoptic side here.  Just to make sure we understand the current procedure:

1) MADIS generates the SNOTEL SHEF files at https://madis-data.ncep.noaa.gov/madisPublic1/data/snotel/hourly/

2) Synoptic Data/MesoWest then ingest the SNOTEL data from those files.

3) MADIS does not use the SHEF files directly to ingest the SNOTEL data, but instead relies upon the public IOOS-CSV format generated by Synoptic Data PBC (mesowest_csvplus.dat.gz) to ingest the SNOTEL data instead of the SHEF files?

If we have that correct, then I believe the issues do reside originally with the issue of the SHEF files not always having all of the data during the overnight hours.  We provided an example back on December 12 illustrating what we had found with the SHEF files during the time period of UTC midnight to midnight local time (which would match the 00-09Z timeframe in the original request).  Let me know if resending that document might be helpful.

The mesowest_csvplus.dat.gz file we transmit has a temporal limit on how far back it queries for data, in order to keep the volume/file size down in the 5-minute sending to MADIS, so backfilled SNOTEL temperature data would not be in that file being transmitted if outages in the SNOTEL SHEF files are lasting several hours each night.

Happy to answer any other questions as well.

Regards

-Alex


---- on Thu, 04 Jan 2024 12:36:44 -0500 "Leon Benjamin - NOAA Affiliate"<address@hidden> wrote ----
Dale,

Sorry for you getting involved in this.
 
This is an Alaskan SNOTEL site.  It needs SYNOPTIC support not the SYNOPTIC subproviders support.

-- Leon

On Thu, Jan 4, 2024 at 10:32 AM Dale Kirmer [KDOT] <address@hidden> wrote:

Hi,

I am assuming this site description is looking for RWIS data from Kansas?

We had something interrupt our RWIS data from January 4, 2024 at 12:00 am till 9:35am.

We should be back up and running.

 

Best Regards,

Dale Kirmer

 

Dale Kirmer, P.E. | Staff Engineer, Bureau of Maintenance

O: 785.296-6355

address@hidden

 

 

From: Leon Benjamin - NOAA Affiliate <address@hidden>
Sent: Thursday, January 04, 2024 11:27 AM
To: MITCH <address@hidden>
Cc: Support User <address@hidden>; _NWS NCEP MADIS-data <address@hidden>
Subject: Re: SNOTEL site TUGA2 temperature data issue Jan 03

 

EXTERNAL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe.

Mitch,

This is a site that SYNOPTIC pulls the data from SHEF files at: https://madis-data.ncep.noaa.gov/madisPublic1/data/snotel/hourly/

 

SYNOPTIC processes those files and includes data from them in the IOOS file that it ldms to MADIS.

 

Who understands that data feed and process? 

 

-- Leon

 

On Thu, Jan 4, 2024 at 10:22 AM MITCH <address@hidden> wrote:

Leon,

       Unsure, and not my site.  However,  I did notice yesterday that my site in Calcium NY was not reporting out.  I assumed it was a firewall issue on my end. 

On Jan 4, 2024, at 11:37 AM, Leon Benjamin - NOAA Affiliate <address@hidden> wrote:

All,

 

Can you look through your records and see if you sent MADIS TUGA2 temperature data for each hour from Jan 03 00z to 09z.

 

We had a question from the field and I'm trying to track down where we had a problem.

 

Unfortunately, the SNOTEL SHEF file doesn't go back enough for me to check, but I do see data in the logs for the SHEF file. 

 

Thanks,

-- Leon

 

Leon Benjamin

MADIS Tier 3 support

MADIS help email: address@hidden

 


 

--

Leon Benjamin

MADIS Tier 3 support

MADIS help email: address@hidden

 



--
Leon Benjamin
MADIS Tier 3 support
MADIS help email: address@hidden





--
Leon Benjamin
MADIS Tier 3 support
MADIS help email: address@hidden



--
Leon Benjamin
MADIS Tier 3 support
MADIS help email: address@hidden
MADIS Web page: https://madis.ncep.noaa.gov/