Compatibility Test Suite

This is a suite of tests that are available and that you can run on your Syntax Parser and Renderer to verify that they work just fine.

Adding a new Test Suite

Imagine that you've developed a new Syntax that is named mysyntax/1.0. To test it you'd write the following unit test:

Running it the first time will generate the following type of JUnit results (example is based on XWiki Syntax 1.0, just replace it mentally with "My Syntax 1.0"):

As you can see all the tests are marked as "Missing" since you haven't written them yet.

Adding Tests

The first thing to do is to create a src/test/resources/syntax10 directory that will hold your tests.

Then check the list of missing tests and for each missing test you need to write a test in the corresponding directory. For example for simple/bold/bold1 you'll need to add a test file in src/test/resources/syntax10/simple/bold/.

There are 2 types of tests you can write:

• Tests for your Parser
• Tests for your Renderer

For Parser tests, the format of the files is <test name>.in.txt (e.g. bold1.in.txt)
For Renderer tests, the format of the files is <test name>.out.txt (e.g. bold1.out.txt)

If you have both a Parser and a Renderer and the content of their input and output files are the same then you can merge them into a single file named <test name>.inout.txt (e.g. bold1.inout.txt).

For Parsers tests your goal is to write an input file that generates an XDOM that matches the output expected by the corresponding CTS Test
For Renderer tests, a CTS input file is Parsed and then the resulting XDOM is rendered using your Renderer and the results are compared.

For example here's what you could have (example based  on XWiki Syntax 1.0 which only has a Parser and no Renderer):

Test Configuration

Notice the config.properties file. It contains configuration data for your tests. Namely it can contain the following:

• List of tests that are not applicable for your Syntax. These tests are not executed. Example:
  notApplicableTests = .*bold2.*
  notApplicableTests = .*list1.*
• List of failing tests. These tests are ignored but marked as failing when executed. Example:
  failingTests = .*bold2.*
  failingTests = .*list1.*
• Inherit Syntax. If your Syntax extends an existing syntax you might want to run all the tests for that other syntax too, in this case you can specify an inherit syntax like this (in this example we inherit from XWiki Syntax 2.0):
  inheritSyntax = xwiki/2.0

Note that notApplicableTests and failingTests are specified as regexes against the name of the tests as you see when you execute them. For example: simple/bold/bold1 [xwiki/2.0, IN:bold1.in.txt, CTS:bold1.inout.xml].

Generating a Report

We're providing a nice CTS Report in the form of a code snippet that you can copy/paste in you own wiki instance.

Since we're running this Compatibility Test Suite on all the Syntaxes that we support you can checkout the resulting Test Report.

Advanced Stuff

• If your Parser generates an XDOM with Macros that contain macros with content in a different Syntax that yours then you can specify this in the @Syntax annotation you use. For example the XWiki Syntax 1.0 Parser converts XWiki Syntax 1.0 content into XWiki Syntax 2.0 content and thus its Test Suite is written like this:
  @RunWith(CompatibilityTestSuite.class)
  @Syntax(value = "xwiki/1.0", metadata = "xwiki/2.0")
  public class XWiki10CompatibilityTest
  {
  }