Skippable tests

[marionebl]

While working #306 I noticed that it might be beneficial for learners to skip single tests, enabling a workflow where learners implement and enable one test after the other (as seen e.g. on the Rust and F# track)

Also skipping all tests by default might improve the overall track setup as executing dune runtest from the track root can serve as a basic sanity check for all exercises and test harnesses.

An example of how this might look like can be found here

brainpower-org/ocaml-exercism-cubicle/acronym/test.ml#L4-L6

I also thought it might be nice to have a CI step that copies examples.ml and runs the (then enabled) unit tests against it. I'm not sure what the best way to do this would be, thoughts?

I tried to contributes this in the test-generator templates but failed to compile the test-generator.

[sshine]

The main drawback of commenting out tests by default is that lazy students will get the impression that their program works because they haven't enabled all tests. At work we have a BAIL_ON_FAIL=1 parameter that, when set, will stop on the first test that fails. Would it not be preferrable to achieve the same workflow of "one test at a time", but avoid manually commenting in tests?

This is handled in the Haskell track with HSpec's configFastFail = True.

I wonder if OUnit2 has an equivalent.

It doesn't look like it.

[sshine]

I thought that maybe

val run_test_tt_main : ?exit:(int -> unit) -> test -> unit

means that we can exit the test suite given a failure.

But trying

 let () =
  run_test_tt_main ("connect tests" >::: tests)
    ~exit:(fun i ->
        print_endline ("I got " ^ string_of_int i ^ "!");
        exit i
    )

still runs all tests.

So I don't think so.

[marionebl]

I thought about a solution along these lines:

let ae ?(skip=false) exp got _test_ctxt  =
  skip_if skip "Skipped";
  assert_equal exp (got ()) 

let () =
  run_test_tt_main ("tests" >::: [
    "skipped - would fail" >:: ae ~skip:true true (fun _ -> false);;
    "not skipped - succeeds" >:: ae ~skip:false true (fun _ -> true);;
  ])

Running this creates output like this:

.S
Ran: 2 tests in: 0.10 seconds.
OK: Cases: 2 Skip: 1