Posted to the Perl QA list, but I figure some of you might want this.
For some strange reason, many folks seems to hate updating test plans. I don't know why this seems like so much work, given what a tiny task it is compared to the bulk of the code, but basically, here's how it works. You change this:
use Test::More tests => 13;To this:
use Test::More tests => 'no_plan'; # tests => 13And when you're done, you change it back and update the test count. Some people think this is too much work, so based upon some vim code [chromatic] had, I wrote the following plugin. This could use a lot of work, but you might want to save this as ~/.vim/plugin/ToggleTestPlan.vim.
if exists( "toggle_test_plan" )
finish
endif
let toggle_test_plan = 1
map ,tp :call ToggleTestPlan()
function ToggleTestPlan()
call SavePosition()
let curr_line = 1
while curr_line <= line("$")
if match(getline(curr_line), 'More\s*tests') > -1
%s/More tests =>/More 'no_plan'; # tests =>/
call RestorePosition()
elseif match(getline(curr_line), 'More\s*''no_plan') > -1
%s/More 'no_plan';\s*# /More /
endif
let curr_line = curr_line + 1
endwhile
endfunction
function SavePosition()
let s:curLine = winline()
let s:curColumn = wincol()
endfunction
function RestorePosition()
exe s:curLine
exe "normal! ".s:curColumn."|"
endfunction
Basically, that maps ,tp to ToggleTestPlan() and toggles your test plan back and forth. If you switch to 'no_plan', it leaves your cursor where it is. If you switch to tests => $num_tests, it puts your cursor on the right line to change the test number. I could add more, but since I'm such a vim scripting newbie, I figure others are better placed to fix other issues. For example, the stuff for saving and restoring position were originally called like this (ganked from another plugin):
callAnd defined like this:
" SaveCursorPosition function!SaveCursorPosition() let s:curLine = winline() let s:curColumn = wincol() endfunction
Why were they defined like that? Who knows? Any explanations and improvements appreciated.
Update: I now know why they were defined like that. After I get home tonight, I'll do some more work and have a much better version of this code available.
Cheers,
Ovid
New address of my CGI Course.
I always just use no_plan.
-- Hofmator
dieing exits with a non-zero exit status and so will be caught by Test::Harness. But it is possible, if unlikely, for a test to exit early in such a way that without a plan Test::Harness won't notice the problem.
I always use a plan more because I want to notice when a different number of tests gets run, such as due to a bug I've introduced into the *.t file or a loop not running the same number of iterations.
Not that I get this whole "ok 6" idea for a test suite. I'd rather output results, include the correct output, and assert that the generated output matches the correct output.
- tye
It can protect you when your tests end unexpectedly. For example, let's say you're running 30 tests. At some point, another programmer on your team writes a bad function which calls [exit]. Your tests might end prematurely and having a test plan catches that. Or maybe you've just updated a CPAN module which calls [exit] when it shouldn't or finds some other way of terminating your tests early. Again, having a test count will protect you. It's quite possible that a test can terminate early without any tests failing.
Another example is when someone does something like this (assumes Scalar::Util::looks_like_number() has been imported):
foreach my $num ( $point->coordinates ) {
ok looks_like_number($num), "... and $num should be a number";
}
If that returns a different number of coordinates from what you expect, having a test plan will catch that. Admittedly, this should actually look something like this:
ok my @coordinates = $point->coordinates,
'coordinates() should return the coordinates';
is scalar @coordinates, 3, '... and it should return the correct number of them';
foreach my $num ( @coordinates ) {
ok looks_like_number($num), "... and each should be a number ($num)";
}
With that, because you're explicitly counting the number of coordinates, the test plan is not as necessary. However, as with the [exit] example, a test plan not only helps out when the code is less than perfect, it also helps out when the tests are less than perfect. It's such a small thing to update and when it actually catches a problem, you'll be greatful.
Some people argue, "yeah, but I never write code that stupid so this doesn't apply to me!". That's fine. If updating the test plan is too much work for them, so be it. Me, I know I make mistakes and I expect others to make mistakes. If we didn't make mistakes, we wouldn't need the tests in the first place.
(Trivia quiz: if the above tests were in a CPAN module, how might those tests fail?)
Cheers,
Ovid
New address of my CGI Course.
I've never been caught out by an errant exit() but rather some kind of fault from bad XS. It looks like the script just exited but in reality something really horrible just occurred.
⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊
A picture is worth a thousand words, but takes 200K.
use Test::More 'declare_done'; # tests here.... done_testing();I've declared that my plan is to declare when I'm done testing. At the end of the test script, I do just that. Now, if the script exits early and done_testing() hasn't be called, we know there was a problem, and we know where it is, because we know the last test that was run.
[markjugg|Mark Stosberg]
Excellent idea. Simplicity incarnate.
... you might want to save this as ~/.vim/plugin/ToggleTestPlan.vim.
Or even better, save it as ~/.vim/ftplugin/perl_toggle_test_plan.vim; using the ftplugin/ directory and putting perl_ at the start of its name will ensure it's only loaded for Perl files.
That would be better written as:map ,tp :call ToggleTestPlan()<cr>
map,tp :call ToggleTestPlan()<cr>
That will make the mapping local to the current buffer, so it won't leak out into other (non-Perl) files you subsequently open in the same Vim session.
Why were they defined like that (with)? Who knows?
Here's the same thing written in Elisp for all those Emacs users out there.
;; Ctrl-C t p
;; (global-set-key "\C-ctp" 'toggle-test-plan)
;; or rather, only set this when editing perl code
(eval-after-load "cperl-mode"
'(add-hook 'cperl-mode-hook
(lambda ()
;; ... other perl only key bindings go here
(local-set-key "\C-ctp" 'toggle-test-plan)
))
(defun toggle-test-plan ()
"..."
(interactive)
(let ((new-pos))
(save-excursion
(goto-char (point-min))
(cond ((re-search-forward "More[ \t]+tests[ \t]*=>[ \t]*" nil t)
(replace-match "More 'no_plan'; # tests => " t t))
((re-search-forward "More[ \t]+'no_plan';[ \t]*#[ \t]*" nil t)
(replace-match "More " t t)
(setq new-pos (point)))))
(if new-pos (goto-char new-pos))))
⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊
Our clever guys at work use the following rule to set up test plans:
use Test::More;
....
my $tests_planned;
...
# subsystem 1 test
BEGIN { $tests_planned ++ }
# carry out one test
...
# subsystem 2 test
BEGIN { $test_planned += 3 }
# carry out our 3 tests
...
...
# Then plan the tests for Test::More right at the end
BEGIN { plan( tests => $tests_planned ); }
This way, the programmers only have to make sure the number of tests planned for a subsystem is true and square, and don't have to worry about the number of planned tests getting out of sync at all.
This isn't actually very lazy.
Besides having to remember to keep the number of BEGIN blocks synchronised with the addition and removal of tests, if you ever do get out of sequence in anything other than the most trivial of tests files, working out which of the BEGIN blocks needs to be adjusted is a total pain in the neck.
This is especially true if someone else made the change that threw the sequencing out.
<bias class="author">
If you do this sort of thing you might be interested in [cpan://Test::Block] where your example would be written as:
use Test::More 'no_plan';
use Test::Block qw( $Plan );
{ local $Plan = 1;
# carry out one test
}
{ local $Plan = 3;
# carry out our 3 tests
}
</bias>
Why can't the module count the tests before running them? When running tests in other languages, like Ruby, or Java, or what have you, you never have to give a test count and it's no problem there. Seems like a good kind of laziness to me - and a way to avoid mistakes automatically.
foreach my $property ( $object->property ) {
ok defined $property, "... and $property should be defined";
}
How many tests is that?
Cheers,
Ovid
New address of my CGI Course.
Yes good point, as it was formulated. That was clumpsy.
But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect. I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12). Note that this does not mean you have to have a teardown, that's in the base class. Also, they seem to catch any exits and other stuff, so you can get a pass, a fail or an error out of a test, and they usually keep on running the rest. Also quite easy to track.
Is it not possible to have a runner of that kind in Perl, or is it just that everyone sticks with the old stuff? It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.
But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect.
Not so easy :-)
Normal exceptions, exiting with a non-zero value, etc. are detected. Unexpected crashes from XS code, POSIX::_exit, or exit(0) are not. See below for why.
. I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12).
It's not really that aspect that causes the problem. Indeed Perl has a couple of testing frameworks that use the xUnit model (Test::Unit and my Test::Class).
The issue is that in frameworks like JUnit, TestNG, etc. the test runner and the tests are running in the same process. If the tests exit for an unexpected reason then nothing runs - so hard crashes of the system are trivial to detect (hey - my test suite has crashed!)
In Perl the test runner (Test::Harness) and the tests (the *.t files) run in separate processes.
This is good in many ways, because it makes our tests more flexible (e.g. we can run tests in different programming languages) and has the advantage that a hard crash will not stop all the other tests running.
The disadvantage is that the test runner has to figure out for itself whether that test script exited because all the tests had run, or whether it exited because something unexpected happened.
One way it does this is via the fixed plan. Double checking that the number of tests that were expected to run actually ran.
It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.
The win is that we have a more flexible testing framework by splitting the test runner and the test producer up.
The cost is that detecting unexpected failures is harder. There's been some conversation on the perl-qa list recently on just this topic, with some ideas that might make the 'no_plan' option even safer.