This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hello all, Sean is correct that CrawlableDatasetAmazonS3 first downloads the file from S3 before performing any operations on it. The reason for that is THREDDS assumes a dataset is on the local filesystem and that it can randomly access the content within. Maybe THREDDS could use HTTP range requests (or similar) to efficiently access remote datasets, but that hasn't been attempted and would probably require a major rewrite. In any event, I consider CrawlableDatasetAmazonS3 "experimental" at this point. In addition to the issue Sean linked to, it doesn't yet scour the S3Objects that it downloads: https://github.com/Unidata/thredds/issues/579 Does s3fs-fuse perform well? I'd be surprised if it did. -Christian Ticket Details =================== Ticket ID: CYK-168267 Department: Support THREDDS Priority: Normal Status: Open =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.