Background
In our original tests for with ESPA surface reflectance and data accessed via s3fs-fuse the performance and latency were very poor which led us to using direct EFS container mounts with duplicated auxiliary data in hls-orchestration.
Recently in container modernization discussions, we've planned to make containers completely isolated so that relevant auxiliary data must be downloaded during container execution. This is feasible for individual daily auxiliary files, but @madhuksridhar pointed out that there are a suite of other ancillary files required by ESPA surface reflectance.
With the recent release of mountpoint-s3 we may have a viable alternative for directly mounting buckets as container filesystems. This would work well for simplifying our new nextgen pipelines. Additionally, we could create a public auxiliary data bucket that science users executing the container locally could access in order to have simple access to the full suite of auxiliary data without need to manage it on their own.
Background
In our original tests for with ESPA surface reflectance and data accessed via s3fs-fuse the performance and latency were very poor which led us to using direct EFS container mounts with duplicated auxiliary data in hls-orchestration.
Recently in container modernization discussions, we've planned to make containers completely isolated so that relevant auxiliary data must be downloaded during container execution. This is feasible for individual daily auxiliary files, but @madhuksridhar pointed out that there are a suite of other ancillary files required by ESPA surface reflectance.
With the recent release of mountpoint-s3 we may have a viable alternative for directly mounting buckets as container filesystems. This would work well for simplifying our new nextgen pipelines. Additionally, we could create a public auxiliary data bucket that science users executing the container locally could access in order to have simple access to the full suite of auxiliary data without need to manage it on their own.