CASSANDRA-21216/CASSANDRA-21260: Clear savedBuffer and savedNextKey in BTree.FastBuilder.reset()#4746
Open
andresbeckruiz wants to merge 2 commits intoapache:cassandra-4.1from
Conversation
…-21260) Two distributed tests that reproduce CASSANDRA-21260 purely through CQL operations and controlled schema disagreement: 1. testSchemaDisagreementCorruptsPartitionViaFastBuilder: Reproduces in-memory BTree corruption where stale ColumnMetadata from a failed READ_REQ deserialization leaks into a Row BTree during mutation, causing ClassCastException. Uses wide tables (~4200 columns) so READ_REQ exceeds the 64KB large-message threshold and is deserialized on SEPWorker threads. 2. testSmallMessageContaminatesSSTableHeaderViaNettyEventLoop: Reproduces SSTable header contamination via small messages on the Netty event loop. Uses a 150-column source table and 2000-column victim table, both under the large-message threshold. Thread reuse is guaranteed by Netty's channel-to-EventLoop binding. Both tests block schema propagation to create a disagreement window, then trigger the FastBuilder pooling bug where reset() fails to clear savedBuffer/savedNextKey, allowing stale state to leak across tables.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BTree.FastBuilder.reset()does not clearsavedBufferorsavedNextKey, allowing staleColumnMetadataobjects to leak when aFastBuilderis reused from the thread-local pool after an exception during message deserialization.During a schema disagreement, a
READ_REQdeserialization failure on a replica leaves aFastBuilderin a dirty state withsavedBufferandsavedNextKeypopulated from the source table'sColumnMetadata. When the same thread reuses thatFastBuilderfor a subsequent BTree construction, the stale entries leak into the newBTree, causing:ColumnMetadataobjects from the source table end up in aRowBTree, causingClassCastException: ColumnMetadata cannot be cast to Rowduring mutations, reads, or flushes. This occurs on the large-message path where messages exceeding ~64KB are deserialized onSEPWorkerthreads that also service mutation tasks.savedBufferleak into a victim table'sSerializationHeadervia deletion-only mutations, writing foreign column entries into the SSTable metadata on disk. This can also occur on the small-message path via Netty event loop thread reuse, lowering the trigger threshold to tables with more than 31 columns.More context regarding these bugs can be found in this discussion thread.
Fix
Null out
savedBufferandsavedNextKeyinFastBuilder.reset()for both leaf and branch BTree nodes. Also addsavedNextKey = nulltoAbstractUpdater.reset()for consistency.Test plan
BTreeFastBuilderContaminationTest:testSchemaDisagreementCorruptsPartitionViaFastBuilder: Wide table (4200 columns) triggers large-message deserialization onSEPWorkerthreads, verifies noClassCastExceptionoccurs after schema disagreement.testSmallMessageContaminatesSSTableHeaderViaNettyEventLoop: Small-message scenario (150 columns) triggers deserialization on Netty event loop, verifies no foreign columns appear in victim SSTable headers.BTreeTest.testFastBuilderResetClearsSavedState: VerifiesFastBuilder.reset()clearssavedBuffer/savedNextKeywhen a builder is abandoned without callingbuild().BTreeTesttests pass (12/12).Patch by Andrés Beck-Ruiz, Runtian Liu, reviewed by <> for CASSANDRA-21216, CASSANDRA-21260
Co-authored-by: Runtian Liu runtian@uber.com