Add dynamic mem_gb estimates and odb-fungi/bac120 dataset configs#16
Open
Add dynamic mem_gb estimates and odb-fungi/bac120 dataset configs#16
Conversation
Previous fixed mem_gb values (5-15 GB per rule) poorly matched actual memory usage, limiting concurrency on small markers while under-reserving for large ones. Now each rule estimates memory from its input files: - provision_alignment: estimates from JSON size × seq_length (JSON is compact mutations; script reconstructs full sequences for all nodes) - branches, trajectories, pairwise: 3× alignment FASTA size - metadata, label_auspice_json: 3× JSON size - train_test_split: 3× branches TSV size With --resources mem_gb=160, this allows ~25 concurrent small markers (1 GB each) vs the old limit of 10 (15 GB each), while correctly reserving 12-46 GB for large markers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- defaults/odb-fungi.yaml: 1,113 OrthoDB fungal markers (24.5k genomes, 23M tips) - defaults/bac120-actinomycetota.yaml: 116 markers (49k genomes) - defaults/bac120-bacillota.yaml: 122 markers (122k genomes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
trajectory.py: 3x → 10x + 5GB base (observed 12x peak on 2.3GB alignments) pairwise: 3x → 6x (observed ~5x peak) _mem_gb: add base parameter for fixed overhead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
rchikhi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New configs
Context
At 122k-taxa scale (Bacillota) and 24k-taxa scale (odb fungi), fixed mem_gb values are both too conservative for small markers and too optimistic for large ones. Observed:
Test plan