Add megatests helper by maxulysse · Pull Request #69 · nf-core/nft-utils

maxulysse · 2026-05-19T17:29:52Z

No description provided.

maxulysse · 2026-05-19T17:49:40Z

@nf-core-bot fox linting pretty please 🙏

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

Copilot

Pull request overview

This PR introduces a new file-listing helper intended for “megatests” style outputs, aiming to make filesystem snapshots work consistently across local directories and cloud URIs by leveraging Nextflow’s filesystem layer when available.

Changes:

Added getAllFiles() to list files under a root path with include/ignore filtering and optional directory entries, with Nextflow-aware path resolution for cloud URIs.
Added/updated nf-test coverage and fixtures for local file listing and an S3-based snapshot scenario, plus related nf-test/Nextflow configuration.
Documented the new helper and added an nf-test plugin dependency for VCF hashing in snapshots.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/nextflow.config	Adds AWS client config (anonymous access) for tests.
tests/getAllFilesFromS3/main.nf.test.snap	New snapshot for S3 listing + downloaded-file checksums.
tests/getAllFilesFromS3/main.nf.test	Adds nf-test that lists an S3 prefix and validates snapshots.
tests/getAllFilesFromS3/main.nf	Adds stub workflow for the S3 test case.
tests/getAllFiles/main.nf.test.snap	New snapshot for `getAllFiles()` local-path behavior.
tests/getAllFiles/main.nf.test	Adds nf-test exercising `getAllFiles()` options (ignore/include/includeDir/ignoreFile).
tests/getAllFiles/main.nf	Adds workflow that generates stable/unstable outputs for listing tests.
tests/getAllFiles/.nftignore	Adds ignore globs used by the new local listing test.
tests/filterNextflowOutput/main.nf.test.snap	Updates snapshots for Nextflow output changes and tool versions.
tests/filterNextflowOutput/main.nf	Minor formatting change in params block.
src/main/java/nf_core/nf/test/utils/Methods.java	Implements `getAllFiles()` + Nextflow path resolution; adds S3 CLI helpers and download helper.
nf-test.config	Loads `nft-vcf` plugin for VCF checksum functionality in tests.
docs/usage.md	Documents `getAllFiles()` usage and options.

Comments suppressed due to low confidence (1)

src/main/java/nf_core/nf/test/utils/Methods.java:1304

downloadFromS3(...) waits for the aws s3 cp process without consuming stdout/stderr (and with redirectErrorStream(false)). The AWS CLI typically writes progress to stderr, which can fill the buffer and deadlock/hang the process. Prefer Utils.runProcess(...) (captures stdout/stderr) or redirect/drain streams, and include stderr in the failure exception to make CI failures diagnosable.

    List<String> cmd = new ArrayList<>(Arrays.asList("aws", "s3", "cp"));
    if (noSignRequest) {
      cmd.add("--no-sign-request");
    }
    cmd.add(s3Uri);
    cmd.add(destFile.toString());

    ProcessBuilder pb = new ProcessBuilder(cmd);
    pb.redirectErrorStream(false);
    Process process = pb.start();
    int exitCode = process.waitFor();
    if (exitCode != 0) {
      throw new IOException(
          "AWS CLI returned exit code " + exitCode + " when downloading: " + s3Uri);
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    List<String> ignoreGlobs =
+        (List<String>) options.getOrDefault("ignore", new ArrayList<String>());
+    List<String> includeGlobs =
+        (List<String>) options.getOrDefault("include", Arrays.asList("**", "*"));
+    Boolean includeDir = (Boolean) options.getOrDefault("includeDir", false);
+    String ignoreFilePath = (String) options.get("ignoreFile");
+
+    List<String> allIgnoreGlobs = new ArrayList<>(ignoreGlobs);
+    if (ignoreFilePath != null && !ignoreFilePath.isEmpty()) {
+      allIgnoreGlobs.addAll(readGlobsFromFile(ignoreFilePath));
+    }


+   */
+  private static Path resolveNextflowPath(String pathString) {
+    try {
+      Class<?> cls = Class.forName("nextflow.file.FileHelper");
+      Method method = cls.getMethod("asPath", String.class);
+      return (Path) method.invoke(null, pathString);
+    } catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException e) {


+    } catch (ClassNotFoundException | NoSuchMethodException | IllegalAccessException e) {
+      return Paths.get(pathString);
+    } catch (InvocationTargetException e) {
+      Throwable cause = e.getCause();
+      throw new RuntimeException(
+          "Failed to resolve path '" + pathString + "': " + cause.getMessage(), cause);
+    }


+    cmd.add(normalizedPath);
+    ProcessBuilder pb = new ProcessBuilder(cmd);
+    pb.redirectErrorStream(false);
+    Process process = pb.start();
+
+    BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
+    List<String> files = new ArrayList<>();
+    String line;


+    // Derive a stable local destination: <tmpdir>/nft-utils-s3/<key>
+    String keyPart = s3Uri.substring("s3://".length()); // "bucket/path/to/file"
+    int firstSlash = keyPart.indexOf('/');
+    String relativeKey = firstSlash >= 0 ? keyPart.substring(firstSlash + 1) : keyPart;
+
+    Path destFile = Paths.get(System.getProperty("java.io.tmpdir"), "nft-utils-s3", relativeKey);


+  /**
+   * Lists all files at the given path, returning sorted relative paths.
+   *
+   * <p>Transparently supports local paths <em>and</em> any cloud URI recognised by
+   * Nextflow (e.g. {@code s3://}, {@code gs://}, {@code az://}) by delegating path
+   * resolution to {@code nextflow.file.FileHelper.asPath()} when that class is
+   * available on the runtime classpath (as it is when running inside nf-test).
+   * Falls back to {@link java.nio.file.Paths#get} for plain local paths when
+   * Nextflow is not present.
+   *


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

nvnieuwk · 2026-05-21T07:31:18Z

This file is getting pretty big, might be worth it to split it up in a future PR

Copilot

Pull request overview

Copilot reviewed 9 out of 11 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

docs/usage.md:231

This example passes noSignRequest to downloadFromS3, but that option is currently ignored by the implementation. Removing it keeps the example aligned with actual behavior.

        path(downloadFromS3("s3://my-bucket/results/${relPath}", noSignRequest: true).toString())

+    // Derive a stable local destination mirroring the key path: <tmpdir>/nft-utils-cloud/<key>
+    int schemeEnd = cloudUri.indexOf("://");
+    String withoutScheme = schemeEnd >= 0 ? cloudUri.substring(schemeEnd + 3) : cloudUri;
+    int firstSlash = withoutScheme.indexOf('/');
+    String relativeKey = firstSlash >= 0 ? withoutScheme.substring(firstSlash + 1) : withoutScheme;


+    List<String> cmd = Arrays.asList("nextflow", "fs", "cp", cloudUri, destFile.toString());
+    ProcessBuilder pb = new ProcessBuilder(cmd);
+    pb.redirectErrorStream(false);
+    Process process = pb.start();
+    int exitCode = process.waitFor();


+    ProcessBuilder pb = new ProcessBuilder(cmd);
+    pb.redirectErrorStream(false);
+    Process process = pb.start();


+    }
+
+    int exitCode = process.waitFor();
+    if (exitCode != 0) {


+
+Downloads a single file from S3 to a temporary local directory and returns the local path. The destination mirrors the S3 key structure under a plugin-specific temp directory, so repeated calls for the same URI are idempotent.
+
+This function requires the AWS CLI to be available on the path.


+Supported named parameters:
+
+| Option          | Type      | Default | Description                                                           |
+| --------------- | --------- | ------- | --------------------------------------------------------------------- |
+| `noSignRequest` | `Boolean` | `false` | Pass `--no-sign-request` to the AWS CLI for publicly readable buckets |
+


+
+```groovy
+// Download a file from a public bucket
+def local_file = downloadFromS3("s3://my-public-bucket/data/sample.vcf.gz", noSignRequest: true)


nvnieuwk

Looks good! Will definitely need to be properly tested once released but I don't see any obvious issues

maxulysse added 2 commits May 19, 2026 19:29

megatests helper

420f146

Update test.sh

7497ccc

maxulysse added 4 commits May 20, 2026 09:53

no unstable files

184b17f

clean up

a9af8c5

prek

cf77d58

improve tests

54f860d

maxulysse commented May 20, 2026

View reviewed changes

Comment thread nf-test.config Outdated

Apply suggestions from code review

fe68a9a

Co-authored-by: Maxime U Garcia <max.u.garcia@gmail.com>

maxulysse requested a review from Copilot May 20, 2026 13:49

Copilot started reviewing on behalf of maxulysse May 20, 2026 13:50 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Potential fix for pull request finding

710f3ca

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

nvnieuwk reviewed May 21, 2026

View reviewed changes

Comment thread src/main/java/nf_core/nf/test/utils/Methods.java Outdated

nvnieuwk reviewed May 21, 2026

View reviewed changes

Comment thread src/main/java/nf_core/nf/test/utils/Methods.java Outdated

maxulysse added 4 commits May 21, 2026 14:10

simplify code

12b19b5

cleanup

331d8ec

better docs

9bdd1fa

prek

da12027

maxulysse requested a review from Copilot May 22, 2026 09:50

Copilot started reviewing on behalf of maxulysse May 22, 2026 09:51 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

maxulysse requested a review from nvnieuwk May 22, 2026 10:52

nvnieuwk approved these changes May 22, 2026

View reviewed changes

maxulysse merged commit 107f7f3 into nf-core:main May 22, 2026
10 checks passed

maxulysse deleted the mega-tests branch May 22, 2026 11:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add megatests helper#69

Add megatests helper#69
maxulysse merged 12 commits into
nf-core:mainfrom
maxulysse:mega-tests

maxulysse commented May 19, 2026

Uh oh!

maxulysse commented May 19, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

nvnieuwk May 21, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

nvnieuwk left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		Downloads a single file from S3 to a temporary local directory and returns the local path. The destination mirrors the S3 key structure under a plugin-specific temp directory, so repeated calls for the same URI are idempotent.

		This function requires the AWS CLI to be available on the path.

Conversation

maxulysse commented May 19, 2026

Uh oh!

maxulysse commented May 19, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

nvnieuwk May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

nvnieuwk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants