Skip to content

Add unit tests for benchmarking metrics and document metric_R2 fallback#3888

Open
tanmaydimriGSOC wants to merge 2 commits intoPecanProject:developfrom
tanmaydimriGSOC:add-metric-tests
Open

Add unit tests for benchmarking metrics and document metric_R2 fallback#3888
tanmaydimriGSOC wants to merge 2 commits intoPecanProject:developfrom
tanmaydimriGSOC:add-metric-tests

Conversation

@tanmaydimriGSOC
Copy link
Copy Markdown

Summary

While exploring the benchmarking module for my GSoC 2026 proposal (validation framework project), I noticed that the core metric functions (metric_RMSE, metric_MAE, metric_cor, metric_R2) had no unit test coverage. This PR adds tests for all four functions and documents a previously undocumented behavior discovered while writing them.

Changes

tests/testthat/test-metrics.R (new file)

  • 9 unit tests covering metric_RMSE, metric_MAE, metric_cor, and metric_R2
  • Tests cover: perfect predictions, known values, NA handling, and edge cases
  • Includes a test that explicitly captures the warning triggered by metric_R2's lm() fallback path

R/metric_R2.R (modified)

  • Added @details section documenting the silent fallback behavior:
    when model output is constant, the correlation formula returns
    NA and the function switches to lm()-based R-squared without notifying the user
  • Added inline comment on the fallback block for future developers
  • This behavior was discovered through the tests and is relevant
    for multi-site validation use cases where silent inconsistency across sites could produce misleading results

Notes

The lm() fallback in metric_R2 triggers an "essentially perfect fit: summary may be unreliable" warning from stats::summary.lm
in edge cases. This is now documented but not changed — a follow-up could add an explicit warning or input check for
constant model output.

Tests were verified locally by sourcing metric functions directly due to dependency installation constraints in local environment.

@tanmaydimriGSOC tanmaydimriGSOC marked this pull request as ready for review April 7, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant