[LDM #IDY-368550]: Issue with pqact on FILE -metadata with -overwrite option

Hi Bryan,

> I'm using ldm version 6.13.11, and I found an issue when trying to use a
> pqact pattern like this:
> 
> IDS|DDPLUS    ^FOO BAZ
> FILE  -overwrite -close -metadata /path/to/file
> 
> The resulting file gets truncated.
> 
> --
> 
> Amateur analysis:
> 
> (Note:  The github links below were intended to follow the version
> associated with 6.13.11, but I have to admit I may have messed that up.
> I think you'll get the gist of it though.)
> 
> It seems that when -overwrite is specified, the code acts as if the file
> might already exist and does an lseek() to the beginning of the file:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1332
> 
> Then writes the data using unio_out():
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1247
> 
> As part of unio_out() it notices that -metadata was specified so it writes
> that data first:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1255
> 
> then writes the product itself:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1261
> 
> If all of that succeeded, it continues:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1345
> 
> checks to see if -overwrite was specified again:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1346
> 
> if so, it truncates the file to the size of the product:
> 
> https://github.com/Unidata/LDM/blob/810a920004325d2fe9dc1347791f1332df4acc4c/pqact/filel.c#L1347
> 
> which doesn't include the length of the already-written metadata and ends
> up truncating a part of the end of the product body by an amount equal to
> the length of the metadata.

Your analysis is spot on.

I've created a beta version that should fix this bug, 
<ftp://ftp.unidata.ucar.edu/pub/ldm/beta/ldm-6.13.12.42.tar.gz>.

I suggest trying it and letting me know how it works.

> --
> 
> Also, I'm wondering if in the case of -overwrite it would be safer to
> write a temporary file, then move it into place after the write is
> completed so as to possibly avoid a race condition for external programs
> trying to reading that file.  An external program might get corrupted data
> if it happens to read the file in the midst of a write or between the
> write call and the truncate call.
> 
> --
> 
> Thanks for looking at this, and thanks for the continued support of ldm.

You're welcome. Thank *you* for sending this in.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: IDY-368550
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.