As well as unit tests, written using the JUnit framework, Projog also contains system tests. The system tests are written using a Projog-specific markup and are contained in the comments of Prolog code.

org.projog.test

The projog-test GitHub project contains classes for parsing Projog's system test file format and running the tests. These classes are used for both the system testing and web site documentation generation of the project.

System Tests File Format

System test files contain comments which, using a Projog-specific markup, specify queries and their expected results embedded in standard Prolog code. The system tests provide an efficient, clear and consistent approach to testing that the Projog engine evaluates queries correctly without focusing on how it does it.

The system tests are stored in two places:

1. In text files stored in sub-directories of the project's src/test/prolog directory.
2. Embedded in the comments of the project's Java files alongside the functionality the system tests are intended to test. The system tests are extracted from the Java comments as part of the build process.

The system tests have two purposes:

1. To confirm that the Projog software works as required by comparing the expected results contained in the system tests against the actual results generated when evaluating the queries.
2. To produce the examples contained in the web based Projog manual. For example, the Between.java file gets automatically transformed into the Between.html file as part of the build process.

Examples of how system tests can be specified using comments (i.e. lines prefixed with a %) are:

1. Test that that the query ?- test. succeeds once and no attempt will be made to find an alternative solution:
   %TRUE test
2. Test that that the query ?- test. succeeds once and will fail when an attempt is made to find an alternative solution:
   %TRUE_NO test
3. Test that that the query ?- test. will fail on the first attempt to evaluate it:
   %FAIL test
4. Test that that the query ?- test. will succeed three times and there will be no attempt to evaluate it for a fourth time:
   %?- test
   %YES
   %YES
   %YES
5. Test that that the query ?- test. will succeed three times and will fail when an attempt is made to evaluate it for a fourth time:
   %?- test
   %YES
   %YES
   %YES
   %NO
6. Test that that the query ?- test(X). will succeed three times and there will be no attempt to evaluate it for a fourth time, specifying expectations about variable unification:
   %?- test(X)
   % X=a
   % X=b
   % X=c
   The test contains the following expectations about variable unification:
   • After the first attempt the variable X will be instantiated to a.
   • After the second attempt the variable X will be instantiated to b.
   • After the third attempt the variable X will be instantiated to c.
7. Test that that the query ?- test(X,Y). will succeed three times and will fail when an attempt is made to evaluate it for a fourth time, specifying expectations about variable unification:
   %?- test(X,Y)
   % X=a
   % Y=1
   % X=b
   % Y=2
   % X=c
   % Y=3
   %NO
   The test contains the following expectations about variable unification:
   • After the first attempt the variable X will be instantiated to a and the variable Y will be instantiated to 1.
   • After the second attempt the variable X will be instantiated to b and the variable Y will be instantiated to 2.
   • After the third attempt the variable X will be instantiated to c and the variable Y will be instantiated to 3.
8. Test that that the query ?- repeat(3), write('hello world'), nl. will succeed three times and there will be no attempt to evaluate it for a fourth time, specifying expectations about what should be written to standard output:
   %?- repeat(3), write('hello world'), nl
   %OUTPUT
   % hello world
   %
   %OUTPUT
   %YES
   %OUTPUT
   % hello world
   %
   %OUTPUT
   %YES
   %OUTPUT
   % hello world
   %
   %OUTPUT
   %YES
   The test contains expectations that every evaluation will cause the text hello world and a new-line character to be written to the standard output stream.
9. Test that while evaluating the query ?- repeat(X). an exception will be thrown with a particular message:
   %?- repeat(X)
   %ERROR Expected Numeric but got: NAMED_VARIABLE with value: X
10. The following would be ignored when running the system tests but would be used when constructing the web based documentation to include a link to test.html:
    %LINK test