Writing and running tests
This module provides a testing framework, and is used in the official suite that tests the specification. All its functions emit output conforming to the Test Anything Protocol.
Also see Writing and running tests in Raku.
multi plan(Cool:D :skip-all($reason)!)
multi plan($number_of_tests)
Specify the count of tests -- usually written at the beginning of a test file.
plan 15; # expect to run 15 tests
In subtests, plan is used to specify the count of tests within the subtest.
If a plan is used, it's not necessary to specify the end of testing with done-testing.
You can also provide a :skip-all named argument instead of a test count, to indicate that you want to skip all of the tests. Such a plan will call exit, unless used inside of a subtest.
plan :skip-all<These tests are only for Windows> unless $*DISTRO.is-win;
plan 1;
ok dir 'C:/'; # this won't get run on non-Windows
If used in a subtest, it will instead return from that subtest's Callable. For that reason, to be able to use :skip-all inside a subtest, you must use a sub instead of a regular block:
plan 2;
subtest "Some Windows tests" => sub { # <-- note the `sub`; can't use bare block
plan :skip-all<We aren't on Windows> unless $*DISTRO.is-win;
plan 1;
ok dir 'C:/'; # this won't get run on non-Windows
}
ok 42; # this will run everywhere and isn't affected by skip-all inside subtest
Note that plan with :skip-all is to avoid performing any tests without marking the test run as failed (i.e. the plan is to not run anything and that's all good). Use skip-rest to skip all further tests, once the run has started (i.e. planned to run some tests, maybe even ran some, but now we're skipping all the rest of them). Use bail-out to fail the test run without running any further tests (i.e. things are so bad, there's no point in running anything else; we've failed).
sub done-testing()
Specify that testing has finished. Use this function when you don't have a plan with the number of tests to run. A plan is not required when using done-testing.
It's recommended that the done-testing function be removed and replaced with a plan function when all tests are finalized. Use of plan can help detect test failures otherwise not reported because tests were accidentally skipped due to bugs in the tests or bugs in the compiler. For example:
sub do-stuff {@};
use Test;
ok .is-prime for do-stuff;
done-testing;
# output:
1..0
The above example is where a done-testing fails. do-stuff() returned nothing and tested nothing, even though it should've returned results to test. But the test suite doesn't know how many tests were meant to be run, so it passes.
Adding plan gives a true picture of the test:
sub do-stuff {@};
use Test;
plan 1;
ok .is-prime for do-stuff;
# output:
1..1
# Looks like you planned 1 test, but ran 0
Note that leaving the done-testing in place will have no effect on the new test results, but it should be removed for clarity.
The done-testing function returns False if any test has failed or less tests were run than planned, it returns True otherwise.
multi ok(Mu $cond, $desc = '')
The ok function marks a test as passed if the given $cond evaluates to True. It also accepts an optional description of the test as second argument.
my $response; my $query; ...;
ok $response.success, 'HTTP response was successful';
In principle, you could use ok for every kind of comparison test, by including the comparison in the expression passed to $cond:
sub factorial($x) { ... };
ok factorial(6) == 720, 'Factorial - small integer';
However, where possible it's better to use one of the specialized comparison test functions below, because they can print more helpful diagnostics output in case the comparison fails.
multi nok(Mu $cond, $desc = '')
The nok function marks a test as passed if the given $cond evaluates to False. It also accepts an optional description of the test as second argument.
my $response; my $query; ...;
nok $query.error, 'Query completed without error';
multi is(Mu $got, Mu:U $expected, $desc = '')
multi is(Mu $got, Mu:D $expected, $desc = '')
Marks a test as passed if $got and $expected compare positively with the eq operator, unless $expected is a type object, in which case === operator will be used instead; accepts an optional description of the test as the last argument.
NOTE: the eq operator stringifies its operands, which means is() is not a good function for testing more complex things, such as lists: is (1, (2, (3,))), [1, 2, 3] passes the test, even though the operands are vastly different. For those cases, use is-deeply routine
my $pdf-document; sub factorial($x) { ... }; ...;
is $pdf-document.author, "Joe", 'Retrieving the author field';
is factorial(6), 720, 'Factorial - small integer';
my Int $a;
is $a, Int, 'The variable $a is an unassigned Int';
Note: if only whitespace differs between the values, is() will output failure message differently, to show the whitespace in each values. For example, in the output below, the second test shows the literal \t in the got: line:
is "foo\tbar", "foo\tbaz"; # expected: 'foo baz'# got: 'foo bar'
is "foo\tbar", "foo bar"; # expected: "foo bar"# got: "foo\tbar"
multi isnt(Mu $got, Mu:U $expected, $desc = '')
multi isnt(Mu $got, Mu:D $expected, $desc = '')
Marks a test as passed if $got and $expected are not equal using the same rules as is(). The function accepts an optional description of the test.
isnt pi, 3, 'The constant π is not equal to 3';
my Int $a = 23;
$a = Nil;
isnt $a, Nil, 'Nil should not survive being put in a container';
multi is_approx(Mu $got, Mu $expected, $desc = '')
NOTE: Removed with Rakudo release 2023.09, deprecated in older versions. Use is-approx instead.
multi is-approx(Numeric $got, Numeric $expected, $desc = '')
multi is-approx(Numeric $got, Numeric $expected, Numeric $abs-tol,
$desc = '')
multi is-approx(Numeric $got, Numeric $expected, $desc = '',
Numeric :$rel-tol is required)
multi is-approx(Numeric $got, Numeric $expected, $desc = '',
Numeric :$abs-tol is required)
multi is-approx(Numeric $got, Numeric $expected, $desc = '',
Numeric :$rel-tol is required,
Numeric :$abs-tol is required)
Marks a test as passed if the $got and $expected numerical values are approximately equal to each other. The subroutine can be called in numerous ways that let you test using relative tolerance ($rel-tol) or absolute tolerance ($abs-tol) of different values.
If no tolerance is set, the function will base the tolerance on the absolute value of $expected: if it's smaller than 1e-6, use absolute tolerance of 1e-5; if it's larger, use relative tolerance of 1e-6.
my Numeric ($value, $expected, $abs-tol, $rel-tol) = ...
is-approx $value, $expected;
is-approx $value, $expected, 'test description';
is-approx $value, $expected, $abs-tol;
is-approx $value, $expected, $abs-tol, 'test description';
is-approx $value, $expected, :$rel-tol;
is-approx $value, $expected, :$rel-tol, 'test description';
is-approx $value, $expected, :$abs-tol;
is-approx $value, $expected, :$abs-tol, 'test description';
is-approx $value, $expected, :$abs-tol, :$rel-tol;
is-approx $value, $expected, :$abs-tol, :$rel-tol, 'test description';
When an absolute tolerance is set, it's used as the actual maximum value by which the first and the second parameters can differ. For example:
is-approx 3, 4, 2; # success
is-approx 3, 6, 2; # fail
is-approx 300, 302, 2; # success
is-approx 300, 400, 2; # fail
is-approx 300, 600, 2; # fail
Regardless of values given, the difference between them cannot be more than 2.
When a relative tolerance is set, the test checks the relative difference between values. Given the same tolerance, the larger the numbers given, the larger the value they can differ by can be.
For example:
is-approx 10, 10.5, :rel-tol<0.1>; # success
is-approx 10, 11.5, :rel-tol<0.1>; # fail
is-approx 100, 105, :rel-tol<0.1>; # success
is-approx 100, 115, :rel-tol<0.1>; # fail
Both versions use 0.1 for relative tolerance, yet the first can differ by about 1 while the second can differ by about 10. The function used to calculate the difference is:
|value - expected|
rel-diff = ────────────────────────
max(|value|, |expected|)
and the test will fail if rel-diff is higher than $rel-tol.
is-approx $value, $expected, :rel-tol<.5>, :abs-tol<10>;
When both absolute and relative tolerances are specified, each will be tested independently, and the is-approx test will succeed only if both pass.
sub is-approx-calculate($got, $expected, $abs-tol where { !.defined or $_ >= 0 },
$rel-tol where { !.defined or $_ >= 0 }, $desc)
This is the actual routine called by is-approx when absolute and relative tolerance are specified. They are tested independently, and the test succeeds only if both pass.
multi is-deeply(Seq:D $got, Seq:D $expected, $reason = '')
multi is-deeply(Seq:D $got, Mu $expected, $reason = '')
multi is-deeply(Mu $got, Seq:D $expected, $reason = '')
multi is-deeply(Mu $got, Mu $expected, $reason = '')
Marks a test as passed if the first and second parameters are equivalent, using the same semantics as the eqv operator. This is the best way to check for equality of (deep) data structures. The function accepts an optional description of the test as the last argument.
use Test;
plan 1;
sub string-info(Str() $_) {
Map.new: (
length => .chars,
char-counts => Bag.new-from-pairs: (
letters => +.comb(/<:letter>/),
digits => +.comb(/<:digit>/),
other => +.comb(/<.-:letter-:digit>/),
))
}
is-deeply string-info('42 Butterflies ♥ Raku'), Map.new((
:21length,
char-counts => Bag.new-from-pairs: ( :15letters, :2digits, :4other, )
)), 'string-info gives right info';
Note: for historical reasons, Seq:D arguments to is-deeply get converted to Lists by calling .cache on them. If you want to ensure strict Seq comparisons, use cmp-ok $got, 'eqv', $expected, $desc instead.
multi cmp-ok(Mu $got is raw, $op, Mu $expected is raw, $desc = '')
Compares $got and $expected with the given $op comparator and passes the test if the comparison yields a True value. The description of the test is optional.
The $op comparator can be either a Callable or a Str containing an infix operator, such as '==', a '~~', or a user-defined infix.
cmp-ok 'my spelling is apperling', '~~', /perl/, "bad speller";
Metaoperators cannot be given as a string; pass them as a Callable instead:
cmp-ok <a b c>, &[!eqv], <b d e>, 'not equal';
A Callable $op lets you use custom comparisons:
sub my-comp { $^a / $^b < rand };
cmp-ok 1, &my-comp, 2, 'the dice giveth and the dice taketh away'
cmp-ok 2, -> $a, $b { $a.is-prime and $b.is-prime and $a < $b }, 7,
'we got primes, one larger than the other!';
multi isa-ok(Mu $var, Mu $type, $desc = "The object is-a '$type.raku()'")
Marks a test as passed if the given object $var is, or inherits from, the given $type. For convenience, types may also be specified as a string. The function accepts an optional description of the test, which defaults to a string that describes the object.
class Womble {}
class GreatUncleBulgaria is Womble {}
my $womble = GreatUncleBulgaria.new;
isa-ok $womble, Womble, "Great Uncle Bulgaria is a womble";
isa-ok $womble, 'Womble'; # equivalent
Note that, unlike isa, isa-ok also matches Roles:
say 42.isa(Numeric); # OUTPUT: «False»
isa-ok 42, Numeric; # OUTPUT: «ok 1 - The object is-a 'Numeric'»
multi can-ok(Mu $var, Str $meth, $desc = "..." )
Marks a test as passed if the given $var can run the method named $meth. The function accepts an optional description. For instance:
class Womble {
method collect-rubbish { ... }
}
my $womble = Womble.new;
# with automatically generated test description
can-ok $womble, 'collect-rubbish';
# => An object of type 'Womble' can do the method 'collect-rubbish'
# with human-generated test description
can-ok $womble, 'collect-rubbish', "Wombles can collect rubbish";
# => Wombles can collect rubbish
multi does-ok(Mu $var, Mu $type, $desc = "...")
Marks a test as passed if the given $var can do the given role $type. The function accepts an optional description of the test.
# create a Womble who can invent
role Invent {
method brainstorm { say "Aha!" }
}
class Womble {}
class Tobermory is Womble does Invent {}
# ... and later in the tests
use Test;
my $tobermory = Tobermory.new;
# with automatically generated test description
does-ok $tobermory, Invent;
# => The object does role Type
does-ok $tobermory, Invent, "Tobermory can invent";
# => Tobermory can invent
sub like(Str() $got, Regex:D $expected, $desc = "text matches $expected.raku()")
Use it this way:
like 'foo', /fo/, 'foo looks like fo';
Marks a test as passed if the first parameter, when coerced to a string, matches the regular expression specified as the second parameter. The function accepts an optional description of the test with a default value printing the expected match.
multi unlike(Str() $got, Regex:D $expected, $desc = "text does not match $expected.raku()")
Used this way:
unlike 'foo', /bar/, 'foo does not look like bar';
Marks a test as passed if the first parameter, when coerced to a string, does not match the regular expression specified as the second parameter. The function accepts an optional description of the test, which defaults to printing the text that did not match.
multi use-ok(Str $code, $desc = "$code module can be use-d ok")
Marks a test as passed if the given $module loads correctly.
use-ok 'Full::Qualified::ModuleName';
Since $code is being turned into an EVAL, you can also pass arguments:
use-ok 'Full::Qualified::ModuleName :my-argument';
multi dies-ok(Callable $code, $reason = '')
Marks a test as passed if the given $code throws an Exception.
The function accepts an optional description of the test.
sub saruman(Bool :$ents-destroy-isengard) {
die "Killed by Wormtongue" if $ents-destroy-isengard;
}
dies-ok { saruman(ents-destroy-isengard => True) }, "Saruman dies";
multi lives-ok(Callable $code, $reason = '')
Marks a test as passed if the given $code does not throw an exception.
The function accepts an optional description of the test.
sub frodo(Bool :$destroys-ring) {
die "Oops, that wasn't supposed to happen" unless $destroys-ring;
}
lives-ok { frodo(destroys-ring => True) }, "Frodo survives";
multi eval-dies-ok(Str $code, $reason = '')
Marks a test as passed if the given $string throws an Exception when evaled as code.
The function accepts an optional description of the test.
eval-dies-ok q[my $joffrey = "nasty";
die "bye bye Ned" if $joffrey ~~ /nasty/],
"Ned Stark dies";
multi eval-lives-ok(Str $code, $reason = '')
Marks a test as passed if the given $string does not throw an exception when evaled as code.
The function accepts an optional description of the test.
eval-lives-ok q[my $daenerys-burns = False;
die "Oops, Khaleesi now ashes" if $daenerys-burns],
"Dany is blood of the dragon";
multi exits-ok($code, $exit, $reason)
Marks a test as passed if the given $code exits with the given $exit code.
The routine accepts an optional description ($reason) for the test.
exits-ok({ say 3; exit 4}, 4, "exited with the right code") # passes
sub throws-like($code, $ex_type, $reason?, *%matcher)
Marks a test as passed if the given $code throws the specific exception expected exception type $ex_type. The code $code may be specified as something Callable or as a string to be EVALed. The exception is specified as a type object.
If an exception was thrown, it will also try to match the matcher hash, where the key is the name of the method to be called on the exception, and the value is the value it should have to pass. For example:
sub frodo(Bool :$destroys-ring) {
fail "Oops. Frodo dies" unless $destroys-ring
};
throws-like { frodo }, Exception, message => /dies/;
The function accepts an optional description of the test as the third positional argument.
The routine makes Failures fatal. If you wish to avoid that, use no fatal pragma and ensure the tested code does not sink the possible Failures. If you wish to test that the code returns a Failure instead of throwing, use fails-like routine instead.
sub fails-not-throws { +"a" }
# test passes, even though it's just a Failure and would not always throw:
throws-like { fails-not-throws }, Exception;
# test detects nothing thrown, because our Failure wasn't sunk or made fatal:
throws-like { no fatal; my $ = fails-not-throws; Nil }, Exception;
Please note that you can only use the string form (for EVAL) if you are not referencing any symbols in the surrounding scope. If you are, you should encapsulate your string with a block and an EVAL instead. For instance:
throws-like { EVAL q[ fac("foo") ] }, X::TypeCheck::Argument;
sub fails-like ( \test where Callable:D|Str:D, $ex-type, $reason?, *%matcher)
Same interface as throws-like, except checks that the code returns a Failure instead of throwing. If the code does throw or if the returned Failure has already been handled, that will be considered as a failed test.
fails-like { +"a" }, X::Str::Numeric,
:message(/'Cannot convert string to number'/),
'converting non-numeric string to number fails';
multi subtest(Pair $what)
multi subtest($desc, &subtests)
multi subtest(&subtests, $desc = '')
The subtest function executes the given block, consisting of usually more than one test, possibly including a plan or done-testing, and counts as one test in plan, todo, or skip counts. It will pass the test only if all tests in the block pass. The function accepts an optional description of the subtest.
class Womble {}
class GreatUncleBulgaria is Womble {
has $.location = "Wimbledon Common";
has $.spectacles = True;
}
subtest {
my $womble = GreatUncleBulgaria.new;
isa-ok $womble, Womble, "Correct type";
is $womble.location, "Wimbledon Common", "Correct location";
ok $womble.spectacles, "Correct eyewear";
}, "Check Great Uncle Bulgaria";
You can also place the description as the first positional argument, or use a Pair with the description and the code in either order. This can be useful for subtests with large bodies.
subtest 'A bunch of tests', {
plan 42;
...
...
}
subtest {
plan 72;
...
...
} => 'Another bunch of tests';
multi todo($reason, $count = 1)
Sometimes tests just aren't ready to be run, for instance a feature might not yet be implemented, in which case tests can be marked as todo. Or it could be the case that a given feature only works on a particular platform - in which case one would skip the test on other platforms.
Mark $count tests as TODO, giving a $reason as to why. By default only one test will be marked TODO.
sub my-custom-pi { 3 };
todo 'not yet precise enough'; # Mark the test as TODO.
is my-custom-pi(), pi, 'my-custom-pi'; # Run the test, but don't report
# failure in test harness.
The result from the test code above will be something like:
not ok 1 - my-custom-pi # TODO not yet precise enough # Failed test 'my-custom-pi' # at test-todo.rakutest line 7 # expected: '3.14159265358979' # got: '3'
Note that if you todo a subtest, all of the failing tests inside of it will be automatically marked TODO as well and will not count towards your original TODO count.
multi skip()
multi skip($reason, $count = 1)
Skip $count tests, giving a $reason as to why. By default only one test will be skipped. Use such functionality when a test (or tests) would die if run.
sub num-forward-slashes($arg) { ... } ;
if $*KERNEL ~~ 'linux' {
is num-forward-slashes("/a/b"), 2;
is num-forward-slashes("/a//b".IO.cleanup), 2;
}
else {
skip "Can't use forward slashes on Windows", 2;
}
Note that if you mark a test as skipped, you must also prevent that test from running.
sub skip-rest($reason = '<unknown>')
Skip the remaining tests. If the remainder of the tests in the test file would all fail due to some condition, use this function to skip them, providing an optional $reason as to why.
my $location; sub womble { ... }; ...;
unless $location ~~ "Wimbledon Common" {
skip-rest "We can't womble, the remaining tests will fail";
exit;
}
# tests requiring functional wombling
ok womble();
# ...
Note that skip-rest requires a plan to be set, otherwise the skip-rest call will throw an error. Note that skip-rest does not exit the test run. Do it manually, or use conditionals to avoid running any further tests.
See also plan :skip-all('...') to avoid running any tests at all and bail-out to abort the test run and mark it as failed.
sub bail-out ($desc?)
If you already know the tests will fail, you can bail out of the test run using bail-out():
my $has-db-connection;
...
$has-db-connection or bail-out 'Must have database connection for testing';
The function aborts the current test run, signaling failure to the harness. Takes an optional reason for bailing out. The subroutine will call exit(), so if you need to do a clean-up, do it before calling bail-out().
If you want to abort the test run, but without marking it as failed, see skip-rest or plan :skip-all('...')
multi pass($desc = '')
The pass function marks a test as passed. flunk marks a test as not passed. Both functions accept an optional test description.
pass "Actually, this test has passed";
flunk "But this one hasn't passed";
Since these subroutines do not provide indication of what value was received and what was expected, they should be used sparingly, such as when evaluating a complex test condition.
multi flunk($reason = '')
The opposite of pass, makes a test fail with an optional message.
sub diag($message)
Display diagnostic information in a TAP-compatible manner on the standard error stream. This is usually used when a particular test has failed to provide information that the test itself did not provide. Or it can be used to provide visual markers on how the testing of a test-file is progressing (which can be important when doing stress testing).
diag "Yay! The tests got to here!";