Feat: add bbmap/clumpify by maxulysse · Pull Request #236 · nf-core/seqinspector

maxulysse · 2026-06-12T11:01:36Z

Replaces Feat: add bbmap clumpify #52 by @erkutilaslan
added clumpify module to the pipeline.
- not sure if any of its outputs goes into multiqc (other than version).
Close add clumpify to seqinspector #41

PR checklist

github-actions · 2026-06-12T11:01:45Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

github-actions · 2026-06-12T11:07:20Z

`nf-core pipelines lint` overall result: Passed ✅

Posted for pipeline commit b4daafd

+| ✅ 201 tests passed       |+
#| ❔   7 tests were ignored |#

Details

❔ Tests ignored:

files_exist - File is ignored: tests/default.nf.test
nextflow_config - Config default ignored: params.fastq_screen_references
nf_test_content - nf_test_content
files_unchanged - File ignored due to lint config: assets/nf-core-seqinspector_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-seqinspector_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-seqinspector_logo_dark.png
multiqc_config - multiqc_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/nf-test.yml
files_exist - File found: .github/actions/get-shards/action.yml
files_exist - File found: .github/actions/nf-test/action.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-seqinspector_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/CONTRIBUTING.md
files_exist - File found: docs/images/nf-core-seqinspector_logo_light.png
files_exist - File found: docs/images/nf-core-seqinspector_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: nf-test.config
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-seqinspector_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowSeqinspector.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config variable (correctly) not found: validation.failUnrecognisedParams
nextflow_config - Config variable (correctly) not found: validation.failUnrecognisedHeaders
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.sample_size= 0.0
nextflow_config - Config default value correct: params.tools_bundle= default
nextflow_config - Config default value correct: params.continue_with_lint_fail= true
nextflow_config - Config default value correct: params.bbmap_clumpify_args= markduplicates=true
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.modules_testdata_base_path= s3://ngi-igenomes/testdata/nf-core/modules/
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_nf_test - '.github/workflows/nf-test.yml' is triggered on expected events
actions_nf_test - '.github/workflows/nf-test.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 25.10.4, Config: 25.10.4
readme - README nf-core template version badge found.
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_if_empty_null - No ifEmpty(null) strings found
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: nf-test.yml
actions_schema_validation - Workflow validation passed: template-version-comment.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix_linting.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
local_component_structure - local subworkflows directory structure is correct 'subworkflows/local/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - BWAMEM2_MEM found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - BBMAP_CLUMPIFY found in conf/modules.config and Nextflow scripts.
modules_config - CHECKQC found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - FQ_LINT found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC_ found in conf/modules.config and Nextflow scripts.
modules_config - PICARD_COLLECTHSMETRICS found in conf/modules.config and Nextflow scripts.
modules_config - SEQFU_STATS found in conf/modules.config and Nextflow scripts.
modules_config - SEQTK_SAMPLE found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 4.0.2
rocrate_readme_sync - RO-Crate descriptions are in sync with README.md.
container_configs - conf/containers_conda_lock_files_amd64.config is up to date
container_configs - conf/containers_conda_lock_files_arm64.config is up to date
container_configs - conf/containers_docker_amd64.config is up to date
container_configs - conf/containers_docker_arm64.config is up to date
container_configs - conf/containers_singularity_https_amd64.config is up to date
container_configs - conf/containers_singularity_https_arm64.config is up to date
container_configs - conf/containers_singularity_oras_amd64.config is up to date
container_configs - conf/containers_singularity_oras_arm64.config is up to date

Run details

nf-core/tools version 4.0.2
Run at 2026-06-17 11:47:48

- Add bbmap/clumpify module with local multiqc_files topic patch - Add nf-test for bbmap/clumpify tool - Update CHANGELOG, CITATIONS, README, docs, schema, and config - Add Erkut Ilaslan as contributor - Configure memory and ext.args for duplication assessment

SPPearce

What will happen with seqtk sample? Does that sample at random, or take the first N?

SPPearce · 2026-06-12T19:41:43Z

+
+</details>
+
+[BBMap Clumpify](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/clumpify-guide/) removes duplicates from sequencing data and creates smaller, faster gzipped FASTQ files. This is particularly useful for reducing file sizes while maintaining data quality.


Suggested change

[BBMap Clumpify](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/clumpify-guide/) removes duplicates from sequencing data and creates smaller, faster gzipped FASTQ files. This is particularly useful for reducing file sizes while maintaining data quality.

[BBMap Clumpify](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/clumpify-guide/) removes duplicates from sequencing data and creates smaller, faster gzipped FASTQ files. This is particularly useful for reducing file sizes while maintaining data quality. Please note that the resulting files will not be random, so tools that take the first X reads will return a biased sample.

So for now, we don't publish the reads from bbmap clumpify, I merely use the tool for duplication assessment. I will update the description

The re-creation of fastq via bbmap clumpify was decided out of scope for now.
cf #41
I will re-assess allowing creation of such files if it could be useful for some users.
It will have to be controlled via parameters obviously

To be honest, probably more useful in demultiplex

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

SPPearce

Ok, if you aren't actually using the fastqs, I'm not worried.

- Fix invalid parameter deduped=true to dedupe=true in test - Add clumped FASTQ output file to docs/output.md - Document bbmap_clumpify_args and save_bbmap_clumpify_reads usage - Add custom tool arguments section to docs/usage.md

maxulysse · 2026-06-17T11:35:13Z

So I agree with Simon's assessment that bbmap clumify might be slightly misplaced here in seqinspector, and might be a better fit in demultiplex.
That being said, I do think this current implementation, which by default works for deduplication assessment, and only produced clumpified reads when using the right params is a good in between solution.
And could be useful for users not using demultiplex, but still wanting some basic qc and fastqc file reduction.

erkutilaslan and others added 5 commits October 29, 2024 10:38

add module bbmap/clumpify

567ea92

cleanup code

541cb3c

fix lint errors

c2fd2d0

fix comment

cd4acf5

Merging dev

50010e0

maxulysse mentioned this pull request Jun 12, 2026

Feat: add bbmap clumpify #52

Closed

11 tasks

maxulysse changed the base branch from master to dev June 12, 2026 11:02

maxulysse added 2 commits June 12, 2026 13:05

Merge branch 'dev' into pr/52

7161580

no tests necessary

922bafb

maxulysse added 3 commits June 12, 2026 13:22

fix patch

cc410ef

bump module

648ef11

maxulysse changed the title ~~Feat: add module bbmap/clumpify~~ Feat: add bbmap/clumpify Jun 12, 2026

SPPearce reviewed Jun 12, 2026

View reviewed changes

maxulysse and others added 5 commits June 16, 2026 15:01

Merge branch 'dev' into pr/52

6e77200

update citation

6c4391d

Update workflows/seqinspector.nf

82d277d

Co-authored-by: Simon Pearce <24893913+SPPearce@users.noreply.github.com>

Enable markduplicates for bbmap clumpify and update docs

feb33bc

no need for these tests

3ea15f6

maxulysse commented Jun 16, 2026

View reviewed changes

Comment thread workflows/seqinspector.nf Outdated

maxulysse added 3 commits June 16, 2026 15:29

Apply suggestions from code review

4c1bfd8

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

reorder dependency list

e9fea82

Add Erkut Ilaslan as contributor

a4ac750

maxulysse requested a review from SPPearce June 16, 2026 13:39

maxulysse commented Jun 16, 2026

View reviewed changes

Comment thread README.md Outdated

maxulysse added 3 commits June 16, 2026 15:40

Apply suggestion from @maxulysse

19ba84e

Fix BBMap URL in toolReferencesMap to match CITATIONS.md

e571fc9

Merge branch 'dev' into pr/52

b4ba42e

SPPearce approved these changes Jun 16, 2026

View reviewed changes

maxulysse added 5 commits June 17, 2026 12:18

update bbmap + snapshot

2ff7324

code polish

937dca8

docs: update BBMap Clumpify TOC description

c525be4

typo

a6f8bb3

remove unstable files from snapshots

b4daafd

SPPearce approved these changes Jun 17, 2026

View reviewed changes

maxulysse merged commit 9ced68d into dev Jun 17, 2026
21 checks passed

maxulysse deleted the pr/52 branch June 17, 2026 12:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: add bbmap/clumpify#236

Feat: add bbmap/clumpify#236
maxulysse merged 27 commits into
devfrom
pr/52

maxulysse commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited by nf-core-bot

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

SPPearce left a comment

Uh oh!

SPPearce Jun 12, 2026

Uh oh!

maxulysse Jun 16, 2026

Uh oh!

maxulysse Jun 16, 2026

Uh oh!

SPPearce Jun 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SPPearce left a comment

Uh oh!

maxulysse commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		</details>

		[BBMap Clumpify](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/clumpify-guide/) removes duplicates from sequencing data and creates smaller, faster gzipped FASTQ files. This is particularly useful for reducing file sizes while maintaining data quality.

Conversation

maxulysse commented Jun 12, 2026

PR checklist

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited by nf-core-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core pipelines lint overall result: Passed ✅

❔ Tests ignored:

✅ Tests passed:

Run details

Uh oh!

SPPearce left a comment

Choose a reason for hiding this comment

Uh oh!

SPPearce Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

maxulysse Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

maxulysse Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

SPPearce Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SPPearce left a comment

Choose a reason for hiding this comment

Uh oh!

maxulysse commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 12, 2026 •

edited by nf-core-bot

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅