Compatibility Test Suite

Last modified by Vincent Massol on 2018/02/21 10:52

This is a suite of tests that are available and that you can run on your Syntax Parser and Renderer to verify that they work just fine.

Adding a new Test Suite

Imagine that you've developed a new Syntax that is named mysyntax/1.0. To test it you'd write the following unit test:

import org.junit.runner.RunWith;
import org.xwiki.rendering.test.cts.CompatibilityTestSuite;
import org.xwiki.rendering.test.cts.Syntax;

@RunWith(CompatibilityTestSuite.class)
@Syntax("mysyntax/1.0")
public class MySyntaxCompatibilityTest
{
}

Running it the first time will generate the following type of JUnit results (example is based on XWiki Syntax 1.0, just replace it mentally with "My Syntax 1.0"):

result1.png

As you can see all the tests are marked as "Missing" since you haven't written them yet.

Adding Tests

The first thing to do is to create a src/test/resources/syntax10 directory that will hold your tests.

Then check the list of missing tests and for each missing test you need to write a test in the corresponding directory. For example for simple/bold/bold1 you'll need to add a test file in src/test/resources/syntax10/simple/bold/.

There are 2 types of tests you can write:

  • Tests for your Parser
  • Tests for your Renderer

For Parser tests, the format of the files is <test name>.in.txt (e.g. bold1.in.txt)
For Renderer tests, the format of the files is <test name>.out.txt (e.g. bold1.out.txt)

If you have both a Parser and a Renderer and the content of their input and output files are the same then you can merge them into a single file named <test name>.inout.txt (e.g. bold1.inout.txt).

For Parsers tests your goal is to write an input file that generates an XDOM that matches the output expected by the corresponding CTS Test
For Renderer tests, a CTS input file is Parsed and then the resulting XDOM is rendered using your Renderer and the results are compared.

For example here's what you could have (example based  on XWiki Syntax 1.0 which only has a Parser and no Renderer):

tests.png

Test Configuration

Notice the config.properties file. It contains configuration data for your tests. Namely it can contain the following:

  • List of tests that are not applicable for your Syntax. These tests are not executed. Example:
    notApplicableTests = .*bold2.*
    notApplicableTests = .*list1.*
  • List of failing tests. These tests are ignored but marked as failing when executed. Example:
    failingTests = .*bold2.*
    failingTests = .*list1.*
  • Inherit Syntax. If your Syntax extends an existing syntax you might want to run all the tests for that other syntax too, in this case you can specify an inherit syntax like this (in this example we inherit from XWiki Syntax 2.0):
    inheritSyntax = xwiki/2.0

Note that notApplicableTests and failingTests are specified as regexes against the name of the tests as you see when you execute them. For example: simple/bold/bold1 [xwiki/2.0, IN:bold1.in.txt, CTS:bold1.inout.xml].

Generating a Report

We're providing a CTS Report in the form of a code snippet that you can copy/paste in you own wiki instance.

Since we're running this Compatibility Test Suite on all the Syntaxes that we support you can checkout the resulting Test Report.

Advanced

Pattern

If you wish your test suite to only execute a single test (to debug only this test for example), you can add the pattern value in the Scope annotation. For example the following will only execute the /table1.test test:

@RunWith(RenderingTestSuite.class)
@RenderingTestSuite.Scope(value = "latex10.specific", pattern="table1.test")
@AllComponents
public class LaTeXSpecificTest
{
}

Mocking

You may wish to perform some initialization of mocks before running the tests. This can be achieved by adding a method annotated with @RenderingTestSuite.Initialized and accepting a MockitoComponentManager parameter, as in the following example:

@RunWith(RenderingTestSuite.class)
@RenderingTestSuite.Scope(value = "latex10.specific")
@AllComponents
public class LaTeXSpecificTest
{
   @RenderingTestSuite.Initialized
   public void initialize(MockitoComponentManager componentManager) throws Exception
   {
        MockSetup.setUp(componentManager);

        componentManager.registerComponent(MockWikiModel.getComponentDescriptor());
   }
}

Also notice the usage of the @AllComponents annotation which tells the Rendering Test Site to automatically register all XWiki Components found in the ClassLoader used to load this test. You could also use the @ComponentList({...}) annotation to only register some components. For example:

@ComponentList({
    DefaultBeanManager.class,
    DefaultConverterManager.class,
    EnumConverter.class,
    ConvertUtilsConverter.class
})

MetaData

If your Parser generates an XDOM with Macros that contain macros with content in a different Syntax that yours then you can specify this in the @Syntax annotation you use. For example the XWiki Syntax 1.0 Parser converts XWiki Syntax 1.0 content into XWiki Syntax 2.0 content and thus its Test Suite is written like this:

@RunWith(CompatibilityTestSuite.class)
@Syntax(value = "xwiki/1.0", metadata = "xwiki/2.0")
public class XWiki10CompatibilityTest
{
}
  • Powered by XWiki 15.10.8-node2. Hosted and managed by XWiki SAS

Get Connected