=head1 NAMETest::Harness::TAP - Documentation for the TAP format=head1 SYNOPSISTAP, the Test Anything Protocol, is Perl's simple text-based interfacebetween testing modules such as Test::More and the test harnessTest::Harness.=head1 TODOExit code of the process.=head1 THE TAP FORMATTAP's general format is:    1..N    ok 1 Description # Directive    # Diagnostic    ....    ok 47 Description    ok 48 Description    more tests....For example, a test file's output might look like:    1..4    ok 1 - Input file opened    not ok 2 - First line of the input valid    ok 3 - Read the rest of the file    not ok 4 - Summarized correctly # TODO Not written yet=head1 HARNESS BEHAVIORIn this document, the "harness" is any program analyzing TAP output.Typically this will be Perl's I<prove> program, or the underlyingC<Test::Harness::runtests> subroutine.A harness must only read TAP output from standard output and notfrom standard error.  Lines written to standard output matchingC</^(not )?ok\b/> must be interpreted as test lines.  All otherlines must not be considered test output.=head1 TESTS LINES AND THE PLAN=head2 The planThe plan tells how many tests will be run, or how many tests haverun.  It's a check that the test file hasn't stopped prematurely.It must appear once, whether at the beginning or end of the output.The plan is usually the first line of TAP output and it specifies howmany test points are to follow. For example,    1..10means you plan on running 10 tests. This is a safeguard in case your testfile dies silently in the middle of its run.  The plan is optional but ifthere is a plan before the test points it must be the first non-diagnosticline output by the test file.In certain instances a test file may not know how many test pointsit will ultimately be running. In this case the plan can be the lastnon-diagnostic line in the output.The plan cannot appear in the middle of the output, nor can it appear morethan once.=head2 The test lineThe core of TAP is the test line.  A test file prints one test line testpoint executed. There must be at least one test line in TAP output. Eachtest line comprises the following elements:=over 4=item * C<ok> or C<not ok>This tells whether the test point passed or failed. It must beat the beginning of the line. C</^not ok/> indicates a failed testpoint. C</^ok/> is a successful test point. This is the only mandatorypart of the line.Note that unlike the Directives below, C<ok> and C<not ok> arecase-sensitive.=item * Test numberTAP expects the C<ok> or C<not ok> to be followed by a test pointnumber. If there is no number the harness must maintainits own counter until the script supplies test numbers again. Sothe following test output    1..6    not ok    ok    not ok    ok    okhas five tests.  The sixth is missing.  Test::Harness will generate    FAILED tests 1, 3, 6    Failed 3/6 tests, 50.00% okay=item * DescriptionAny text after the test number but before a C<#> is the description ofthe test point.    ok 42 this is the description of the testDescriptions should not begin with a digit so that they are not confusedwith the test point number.The harness may do whatever it wants with the description.=item * DirectiveThe test point may include a directive, following a hash on thetest line.  There are currently two directives allowed: C<TODO> andC<SKIP>.  These are discussed below.=backTo summarize:=over 4=item * ok/not ok (required)=item * Test number (recommended)=item * Description (recommended)=item * Directive (only when necessary)=back=head1 DIRECTIVESDirectives are special notes that follow a C<#> on the test line.Only two are currently defined: C<TODO> and C<SKIP>.  Note thatthese two keywords are not case-sensitive.=head2 TODO testsIf the directive starts with C<# TODO>, the test is counted as atodo test, and the text after C<TODO> is the explanation.    not ok 13 # TODO bend space and timeNote that if the TODO has an explanation it must be separated fromC<TODO> by a space.These tests represent a feature to be implemented or a bug to be fixedand act as something of an executable "things to do" list.  They areB<not> expected to succeed.  Should a todo test point begin succeeding,the harness should report it as a bonus.  This indicates that whateveryou were supposed to do has been done and you should promote this to anormal test point.=head2 Skipping testsIf the directive starts with C<# SKIP>, the test is counted as havingbeen skipped.  If the whole test file succeeds, the count of skippedtests is included in the generated output.  The harness should reportthe text after C< # SKIP\S*\s+> as a reason for skipping.    ok 23 # skip Insufficient flogiston pressure.Similarly, one can include an explanation in a plan line,emitted if the test file is skipped completely:    1..0 # Skipped: WWW::Mechanize not installed=head1 OTHER LINES=head2 Bail out!As an emergency measure a test script can decide that further testsare useless (e.g. missing dependencies) and testing should stopimmediately. In that case the test script prints the magic words    Bail out!to standard output. Any message after these words must be displayedby the interpreter as the reason why testing must be stopped, asin    Bail out! MySQL is not running.=head2 DiagnosticsAdditional information may be put into the testing output on separatelines.  Diagnostic lines should begin with a C<#>, which the harness mustignore, at least as far as analyzing the test results.  The harness isfree, however, to display the diagnostics.  Typically diagnostics areused to provide information about the environment in which test file isrunning, or to delineate a group of tests.    ...    ok 18 - Closed database connection    # End of database section.    # This starts the network part of the test.    # Daemon started on port 2112    ok 19 - Opened socket    ...    ok 47 - Closed socket    # End of network tests=head2 Anything elseAny output line that is not a plan, a test line or a diagnostic isincorrect.  How a harness handles the incorrect line is undefined.Test::Harness silently ignores incorrect lines, but will become morestringent in the future.=head1 EXAMPLESAll names, places, and events depicted in any example are whollyfictitious and bear no resemblance to, connection with, or relation to anyreal entity. Any such similarity is purely coincidental, unintentional,and unintended.=head2 Common with explanationThe following TAP listing declares that six tests follow as well asprovides handy feedback as to what the test is about to do. All sixtests pass.    1..6    #    # Create a new Board and Tile, then place    # the Tile onto the board.    #    ok 1 - The object isa Board    ok 2 - Board size is zero    ok 3 - The object isa Tile    ok 4 - Get possible places to put the Tile    ok 5 - Placing the tile produces no error    ok 6 - Board size is 1=head2 Unknown amount and failuresThis hypothetical test program ensures that a handful of servers areonline and network-accessible. Because it retrieves the hypotheticalservers from a database, it doesn't know exactly how many servers itwill need to ping. Thus, the test count is declared at the bottom afterall the test points have run. Also, two of the tests fail.    ok 1 - retrieving servers from the database    # need to ping 6 servers    ok 2 - pinged diamond    ok 3 - pinged ruby    not ok 4 - pinged saphire    ok 5 - pinged onyx    not ok 6 - pinged quartz    ok 7 - pinged gold    1..7=head2 Giving upThis listing reports that a pile of tests are going to be run. However,the first test fails, reportedly because a connection to the databasecould not be established. The program decided that continuing waspointless and exited.    1..573    not ok 1 - database handle    Bail out! Couldn't connect to database.=head2 Skipping a fewThe following listing plans on running 5 tests. However, our programdecided to not run tests 2 thru 5 at all. To properly report this,the tests are marked as being skipped.    1..5    ok 1 - approved operating system    # $^0 is solaris    ok 2 - # SKIP no /sys directory    ok 3 - # SKIP no /sys directory    ok 4 - # SKIP no /sys directory    ok 5 - # SKIP no /sys directory=head2 Skipping everythingThis listing shows that the entire listing is a skip. No tests were run.    1..0 # skip because English-to-French translator isn't installed=head2 Got spare tuits?The following example reports that four tests are run and the last twotests failed. However, because the failing tests are marked as thingsto do later, they are considered successes. Thus, a harness should reportthis entire listing as a success.    1..4    ok 1 - Creating test program    ok 2 - Test program runs, no error    not ok 3 - infinite loop # TODO halting problem unsolved    not ok 4 - infinite loop 2 # TODO halting problem unsolved=head2 Creative libertiesThis listing shows an alternate output where the test numbers aren'tprovided. The test also reports the state of a ficticious board game indiagnostic form. Finally, the test count is reported at the end.    ok - created Board    ok    ok    ok    ok    ok    ok    ok    # +------+------+------+------+    # |      |16G   |      |05C   |    # |      |G N C |      |C C G |    # |      |  G   |      |  C  +|    # +------+------+------+------+    # |10C   |01G   |      |03C   |    # |R N G |G A G |      |C C C |    # |  R   |  G   |      |  C  +|    # +------+------+------+------+    # |      |01G   |17C   |00C   |    # |      |G A G |G N R |R N R |    # |      |  G   |  R   |  G   |    # +------+------+------+------+    ok - board has 7 tiles + starter tile    1..9=head1 AUTHORSAndy Lester, based on the original Test::Harness documentation by Michael Schwern.=head1 ACKNOWLEDGEMENTSThanks toPete Krawczyk,Paul Johnson,Ian Langworthand Nik Claytonfor help and contributions on this document.The basis for the TAP format was created by Larry Wall in theoriginal test script for Perl 1.  Tim Bunce and Andreas Koenigdeveloped it further with their modifications to Test::Harness.=head1 COPYRIGHTCopyright 2003-2005 byMichael G Schwern C<< <schwern@pobox.com> >>,Andy Lester C<< <andy@petdance.com> >>.This program is free software; you can redistribute it and/ormodify it under the same terms as Perl itself.See L<http://www.perl.com/perl/misc/Artistic.html>.=cut