[Pruning] Added a design doc for tlog-tiles pruning#935
[Pruning] Added a design doc for tlog-tiles pruning#935mhutchinson wants to merge 2 commits intotransparency-dev:mainfrom
Conversation
This is a design for transparency-dev#855. If accepted, we can move on to implementation.
|
|
||
| ## Spec Requirements & Ambiguities | ||
|
|
||
| The [tlog-tiles] spec (revision `1cd32db`) defines the rules for pruning but leaves operational details as an exercise for the implementation: |
| * **Ordering**: To maintain consistency with Tessera's existing Garbage Collection mechanism (which cleans up obsolete partial tiles), the tool should process resources in a bottom-up, left-to-right order: first entry bundles, then Level 0 tiles, and finally higher-level tiles. Within each resource type, it should proceed from the lowest index to the highest. | ||
|
|
||
| ### 4. Safety Features | ||
| * **Confirmation Prompt**: The tool scans the range to be pruned and prompts the operator with the count of files (tiles and entry bundles) to be moved before proceeding. |
There was a problem hiding this comment.
Is it necessary to actually list the objects in order to count them, or would it suffice to just calculate the lower bound on the number of resources given the range?
Personally, I think that knowing there are x thousand logical tile/bundle addresses to be moved is more useful than xxx thousand partial and full revisions of some unknown number of logical addresses to be moved.
There was a problem hiding this comment.
I don't much mind TBH. I guess scanning the range is expensive so this needs to be reconsidered. Leaving this comment open.
|
|
||
| ### 4. Safety Features | ||
| * **Confirmation Prompt**: The tool scans the range to be pruned and prompts the operator with the count of files (tiles and entry bundles) to be moved before proceeding. | ||
| * **Undo Command**: The tool provides an `undo` command that moves files back from the archive to the active area and restores the `min_index` file to its previous value (by storing the previous value in the archive during pruning), facilitating recovery from operator error. In the event of manual recovery, the operator can manually update the `min_index` file based on the data they have restored. |
There was a problem hiding this comment.
How does this interact with GC?
There was a problem hiding this comment.
The undo feature? I had assumed that by the time one prunes a tree, it would be thoroughly GCd. Is it a supported configuration to have a no-GC deployment though?
| ### Archive-in-Place (Renaming Files) | ||
| We considered keeping archived files in the same storage location (bucket or directory) but renaming them (e.g., appending `~` or moving to a sub-folder like `archive/` within the same bucket). | ||
|
|
||
| * **Why it was not picked**: While this makes the "move" operation fast and atomic (metadata rename), it creates security risks for cloud storage users. If a GCS bucket is served publicly with uniform bucket-level access, everything in that bucket is public. Hiding an `archive/` folder or renamed files would require complex IAM conditions or putting a proxy/load balancer in front of the bucket. This violates Tessera's goal of zero-infrastructure read paths. Moving to a separate private bucket provides a much stronger security boundary with less operational complexity. |
There was a problem hiding this comment.
You could also just accept that motivated folks would still be able to access archived resources via the (unspecified) archive path?
I'm not sure I understand why a strong security boundary for archived resources is important.
There was a problem hiding this comment.
It's not a strong security boundary. It's:
a) avoiding polluting the serving namespace, e.g. your comment about the min_index file
b) a minor concern about any edge caches also caching this stuff if requested.
If we're strongly opinionated that putting this stuff on the public paths is cool, we can consider changing the primary suggestion for these alternatives.
| We considered relying on cloud-specific features like GCS Object Versioning or Soft Delete. Under this model, the tool would simply delete the files from the public bucket. If Object Versioning is enabled, the deleted files would become noncurrent versions (returning `404` to public requests) and could be restored later. | ||
|
|
||
| * **Why it was not picked**: | ||
| * **Cost Inefficiency**: Tessera frequently updates partial tiles and entry bundles during normal operation. If Object Versioning were enabled on the bucket, every such update would create a new billable version, leading to uncontrolled storage cost growth unless complex lifecycle rules were configured to clean them up. Furthermore, if versioning is enabled, Tessera's existing Garbage Collection mechanism (which deletes obsolete partial tiles) becomes ineffective at saving quota, as the deleted files are merely retained as noncurrent versions. |
There was a problem hiding this comment.
I'm not arguing that object versioning is the right approach at all, but the assertion here about Tessera frequently updating partial tiles and entry bundles is... not true?
Tessera only ever re-writes a tile/entrybundle resource if there was an unrecoverable error during integration, which is relatively rare compared to the number of resources written overall.
And GC could, of course, just explicitly delete the object plus all of its (almost certainly 1) versions.
There was a problem hiding this comment.
Yeah good catch. Updating is the wrong term. This was going in a few different directions and ended up botched. Tried to clarify what was intended: object versioning is not recommended because GC cleans up a lot of stuff. We don't want to start recommending object versioning for this feature, if that then makes GC harder.
| * **Cost Inefficiency**: Tessera frequently updates partial tiles and entry bundles during normal operation. If Object Versioning were enabled on the bucket, every such update would create a new billable version, leading to uncontrolled storage cost growth unless complex lifecycle rules were configured to clean them up. Furthermore, if versioning is enabled, Tessera's existing Garbage Collection mechanism (which deletes obsolete partial tiles) becomes ineffective at saving quota, as the deleted files are merely retained as noncurrent versions. | ||
| * **Portability**: This approach relies on specific cloud provider features that do not translate well to simple POSIX filesystems or other cloud providers with different versioning semantics. The "move to archive" approach is universally applicable across all storage drivers supported by Tessera. | ||
|
|
||
| * **Note**: While we rejected Object Versioning for the reasons above, a variant relying solely on **Soft Delete** (without versioning enabled) is considered as a viable option for GCS deployments (see the **GCS "Brave Mode" Alternative** in the Archiving Strategy section). |
There was a problem hiding this comment.
This seems inconsistent with the opening para in this section, would it make sense to remove Soft Delete entirely here and just leave this dismissal for Object Versioning?
mhutchinson
left a comment
mhutchinson
left a comment
There was a problem hiding this comment.
Thanks for the comments. Addressed many of them.
|
|
||
| The tool performs the following steps: | ||
| 1. **Validation**: Ensures the target index $N$ is less than the current checkpoint size, greater than or equal to the current `min_index`, and enforces that $N$ is a multiple of 256 (to align with tile boundaries). | ||
| 2. **State Commitment**: Writes the new $N$ to a statically served `min_index` file at the root of the log. |
There was a problem hiding this comment.
That was my initial thought, but the spec explicitly has this TODO:
TODO: Some HTTP endpoint for fetching the minimum index? The semantics would be something like: serving a minimum index equivalent to returning 404 from the tiles that would be deleted by the pruning criteria, including when evaluating a log client's availability policies.
If logs are going to do pruning, it seems like a good idea that they at least advertise what they believe they are running with. Even if this isn't cryptographically committed to. Really we need the spec to clarify this, and so I proposed this as a forcing function.
| * **Ordering**: To maintain consistency with Tessera's existing Garbage Collection mechanism (which cleans up obsolete partial tiles), the tool should process resources in a bottom-up, left-to-right order: first entry bundles, then Level 0 tiles, and finally higher-level tiles. Within each resource type, it should proceed from the lowest index to the highest. | ||
|
|
||
| ### 4. Safety Features | ||
| * **Confirmation Prompt**: The tool scans the range to be pruned and prompts the operator with the count of files (tiles and entry bundles) to be moved before proceeding. |
There was a problem hiding this comment.
I don't much mind TBH. I guess scanning the range is expensive so this needs to be reconsidered. Leaving this comment open.
|
|
||
| ### 4. Safety Features | ||
| * **Confirmation Prompt**: The tool scans the range to be pruned and prompts the operator with the count of files (tiles and entry bundles) to be moved before proceeding. | ||
| * **Undo Command**: The tool provides an `undo` command that moves files back from the archive to the active area and restores the `min_index` file to its previous value (by storing the previous value in the archive during pruning), facilitating recovery from operator error. In the event of manual recovery, the operator can manually update the `min_index` file based on the data they have restored. |
There was a problem hiding this comment.
The undo feature? I had assumed that by the time one prunes a tree, it would be thoroughly GCd. Is it a supported configuration to have a no-GC deployment though?
| ### Archive-in-Place (Renaming Files) | ||
| We considered keeping archived files in the same storage location (bucket or directory) but renaming them (e.g., appending `~` or moving to a sub-folder like `archive/` within the same bucket). | ||
|
|
||
| * **Why it was not picked**: While this makes the "move" operation fast and atomic (metadata rename), it creates security risks for cloud storage users. If a GCS bucket is served publicly with uniform bucket-level access, everything in that bucket is public. Hiding an `archive/` folder or renamed files would require complex IAM conditions or putting a proxy/load balancer in front of the bucket. This violates Tessera's goal of zero-infrastructure read paths. Moving to a separate private bucket provides a much stronger security boundary with less operational complexity. |
There was a problem hiding this comment.
It's not a strong security boundary. It's:
a) avoiding polluting the serving namespace, e.g. your comment about the min_index file
b) a minor concern about any edge caches also caching this stuff if requested.
If we're strongly opinionated that putting this stuff on the public paths is cool, we can consider changing the primary suggestion for these alternatives.
| We considered relying on cloud-specific features like GCS Object Versioning or Soft Delete. Under this model, the tool would simply delete the files from the public bucket. If Object Versioning is enabled, the deleted files would become noncurrent versions (returning `404` to public requests) and could be restored later. | ||
|
|
||
| * **Why it was not picked**: | ||
| * **Cost Inefficiency**: Tessera frequently updates partial tiles and entry bundles during normal operation. If Object Versioning were enabled on the bucket, every such update would create a new billable version, leading to uncontrolled storage cost growth unless complex lifecycle rules were configured to clean them up. Furthermore, if versioning is enabled, Tessera's existing Garbage Collection mechanism (which deletes obsolete partial tiles) becomes ineffective at saving quota, as the deleted files are merely retained as noncurrent versions. |
There was a problem hiding this comment.
Yeah good catch. Updating is the wrong term. This was going in a few different directions and ended up botched. Tried to clarify what was intended: object versioning is not recommended because GC cleans up a lot of stuff. We don't want to start recommending object versioning for this feature, if that then makes GC harder.
3 participants
This is a design for #855. If accepted, we can move on to implementation.