Mutation Testing in Model Transformation

Faults in model transformations will produce faults in generated models. The correction of such defective models is expensive, what directly affects the quality of the end product. This is why model transformations have to be correctly tested to maintain product quality. For this, mutation testing is a popular technique.

Mutation testing depends on a suite of mutated model transformations. To fuel the test suite a set of mutated transformations has to be developed. A common way of doing so is by applying mutation operators on the actual model transformation.

Our project covers the implementation of a set of mutation operators in a generic way, so that the mutations can be used with any input model transformation.

Problem

To make mutation testing an effective testing method, a complete set of mutation operators and a large number of mutated model transformations are required. Due to the size of such sets it is too expensive to create them manually. Furthermore, the execution of the different model transformations against the test data is very time-consuming.

Therefore, after the identification of a set of mutation operators, automation is required for the generation of mutated model transformations. As a proof of concept of our approach, we have automated the mutation of model transformations defined in the Atlas Transformation Language, ATL, which is a widespread technology for defining model transformations and provides infrastructure to define transformations on transformations.

Background

A transformation on transformations is called higher-order transformation (HOT). It takes a model transformation as input and produces a model transformation as output. HOTs are the key artefact for our implementation. Since HOTs transform model transformations, they operate on the level of the ATL metamodel. This makes it impossible to reason about types of the actual transformation to mutate, but allows to make mutations absolute generic.

To make a generic mutation specific for a particular scenario and a particular model transformation, a second-order HOT is needed. A second-order HOT takes a HOT as input and produces a HOT as output. In our implementation, they also take the specific transformation to mutate, as well as its input and output metamodels, as input.

 

Approach

To generate mutated model transformations we implemented a set of rules which mutate the original model transformation in different ways, so that the output models produced by these mutants are different from the original output model.

We decided to focus on the following mutation operators (please note that we use concepts specific of the ATL metamodel):

• Delete bindings
• Add bindings
• Change the value of bindings
• Add OutPatternElements
• Change the Class of OutPatternElements
• Delete InPatternElements

For each operator we implemented at least one HOT, and only for some of them it was neccessary to create second-order HOTs too.

Examples

There are 3 major types of transformations:

1. simple HOTs, where only one rule is needed,
2. complex HOTs, where we need to focus also on dependencies, and
3. HOTs with second-order HOTS, where we need information from the given input and output metamodels

In the following we show an implementation example for each of these types.

Example of a simple HOT - Change binding values

To change the value of a binding we need just one rule. In our case we searched for all StringExpressions and changed the value to "hello"

Example of complex HOT with dependencies - Delete an InPatternElement

While changing the value of a binding has no dependencies to other elements of our transformation, deleting an InPatternElement can cause some problems, which must be avoided.

If we delete an InPatternElement that is also used in one or more bindings in an OutPatternElement, we have to delete these referenced bindings too.

If we skip this second step the resulting transformation could be invalid as it would not conform to the ATL metamodel.

Original Rule:

 Rule after deleting the InPatternElement:

   

Therefore, to create a valid mutated transformation, we have to

1. Delete the InPatternElement.
2. Search for all bindings with unreferred variables (which referred to our deleted InPatternElement in the past) and delete them too.

Example of HOT with second order HOT - AddBinding

As said before, the ATL metamodel is not enough to guarantee that the transformation is valid regarding the types appearing in the output and input metamodels. To handle this, the transformation is divided in 2 steps.

The first step is to add a binding with a dummy property name to our OutPatternElement:

 

The second step is to transform this rule and replace the dummy property name with an existing name in the output metamodel.

 

Conclusion and further work

While working on this topic, we had to deal with two major problems:

• Consequences which occur when changing an element of a transformation (for example deleting also referring bindings when deleting an InPatternElement)
• Needed as input metamodels of the transformation to mutate, as well as the transformation to mutate, for second-order HOTs (for example the addition of a reference binding needs also the original ATL transformation as input into the second order HOT)

Further work will consist in the development of tools for automated mutations of model transformations. We first want to keep focusing only on ATL and second on other model transformation languages such as QVT.