Skip to content

Improvements from live testing#6404

Open
nadav-govari wants to merge 2 commits intonadav/pr11from
nadav/pr12
Open

Improvements from live testing#6404
nadav-govari wants to merge 2 commits intonadav/pr11from
nadav/pr12

Conversation

@nadav-govari
Copy link
Copy Markdown
Collaborator

Description

There were a number of observations made when testing.

  1. When deploying >2 compactors, the split cache becomes an impediment rather than an optimization. if the splits are never read, theyre just resident in memory and it doesnt take long to start thrashing. Rather than fix the split cache, remove it (for now. I might come back and fix this down the road)

  2. The merge policy operations only produced one merge per level per call, which resulted in huge split backlogs buildings. We now construct as many merge operations as we can on every tick/

  3. The splits table wasnt sorted the way we needed, so our scans were expensive. Added an index to make this easier.

  4. Got rid of the scan cursor. We can do this naturally with a scan limit and a maturity greater than. We'll always fetch the <= 5000 oldest immature splits. As splits get merged and marked for deletion, the head will move naturally.

How was this PR tested?

Extensive testing on a test cluster that sees meaningful traffic, including a backfill of over 900k immature splits using the compactor architecture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant