Extending an EMF Fuzz Testing Framework with support for UML

Extending an EMF Fuzz Testing Framework

with support for UML

Introduction

Randomly or pseudo-randomly generated data is useful in many application areas including software testing and software security. Using this kind of data as input to test software applications, helps improving their quality by increasing code coverage. In the area of model engineering there, currently, exists only very few frameworks that support the generation of UML (Unified Modeling Language) models. Generating such models would help testing software which expects UML models, with or without applied UML profiles, as input. In this blog we present an extension of an EMF (Eclipse Modeling Framework) Fuzz Testing Framework to support UML model generation including UML profiles.

What is Fuzz testing?           

The idea of Fuzz testing is to test software by using (pseudo-)randomly generated data as input in order to find inputs that cause unexpected behavior of the system under test. Using randomly generated input for testing helps improving software quality by increasing code coverage. Therefore, errors can be detected, which would be hard to find with other software testing techniques or manual input data specification. Reproducibility is a very important property of automated tests because it is necessary to reproduce an error in order to analyse and correct it. Additionally, the costs per found error can be significantly decreased. The generation of pseudo-random input is performed by a so called Fuzz generator.

               

Fuzz testing in Model Engineering

Fuzz Testing can also be applied in the area of model engineering, in which the generated inputs are models. Current Fuzz Testing Frameworks do not support the generation of UML models and UML models including UML Profiles. EclipseSource developed a generic Fuzz Testing Framework, which can generate EMF models, for testing EMF tools. Theoretically, this Fuzz Testing Framework should support generation of UML models as well. This assumption is based on the fact that Ecore is the meta-meta model (M3) used in Eclipse while the EMF-based implementation of UML is located in the metamodel layer (M2). Therefore, UML models should be accepted by the framework. However, there are a few problems which have been the starting point for our project.

Our goals

Current Fuzz Testing Frameworks have problems with generating UML models. Our first goal was, therefore, to extend the EMF Fuzz Testing Framework to work with UML models. Furthermore, our second goal was to extend the framework in such a way, that it also works with UML models including UML Profiles. As an additional goal we wrote a framework documentation to support our development process and help further developers to understand the framework quicker.

The Fuzz Testing Framework

As mentioned before, Fuzz testing is about pseudo-randomly generating input data and running tests with this data. Therefore, a test is not written with concrete input data, but with parameters which control the generation process. Parameters specify, for example, the number of test runs, size of the generated models, number of mutations per test run and the model of which instances should be created.

Therefore, the following components are needed to execute a test with the EMF Fuzz Testing Framework:

• The DataProvider injects different model instances into the test case.
• The JUnit TestRunner repeatedly runs the same test with different data.
• The ModelMutator provides different model instances to the DataProvider.

To specify the parameters in the EMF Fuzz Testing Framework a FuzzConfig file is used.

The FuzzConfig file specifies the following parameters:

• seed: Used for the pseudo-random generation in order to ensure reproducibility.
• count: Specifies the total number of test runs.
• testClass: References the corresponding fuzzy test class.
• id
• minObjectsCount: Specifies the minimum number of objects of the generated model.
• mutationCount: Number of mutations to be performed.
• ePackage: The model mutator generates models with instances of EClasses declared in this package.

Challenges & Results

After setting up the project, we realized that there was only code documentation and no framework description. In order to get familiar with the framework and to help further developers we decided to define a framework description as an additional goal. Next, we started with the implementation by trying to mutate UML2 models instead of Ecore models which failed due to various errors, like missing UML datatypes and wrong return values. One issue we worked on for a rather long time is closely related to the implementation of profile mutations. UML2 uses the class Model for applying and accessing profiles. An adaption which highly eases the implementation of profile mutations was thus to inject the generated model into a framework field of type Model instead of type EObject. We solved this issue by declaring the root object’s EClass (Model) in the fuzzy config file.

After we corrected the errors mentioned, we tried to extend the framework in order to enable the application of UML Profiles. The main part here was to develop custom mutations to apply and unapply the concepts of UML Profiles. The mutations we were working on are therefore:

• ApplyProfileMutation
• UnapplyProfileMutation
• ApplyStereotypeMutation
• UnapplyStereotypeMutation

In ApplyProfileMutation we first of all randomly generate a profile with a random number of stereotypes, having a random number of attributes (with randomly selected types). After that, the metaclasses of the model (to which the mutation should be applyed) are randomly extended by the previously generated stereotypes. This step is required in order to indicate which elements of the model can later have stereotypes applied. Finally, the generated profile is applied to the model. UnapplyProfileMutation randomly selects one already applied profile, and unapplies it. This can easily be done by using UML2’s method unapplyProfile.

                   

ApplyStereotypeMutation is the most interesting but also the most complex mutation. In this mutation, first of all, all elements which have applicable stereotypes have to be identified. We initially concentrated on applying stereotypes to classes, attributes and references. After checking these features for applicable stereotypes, a random number of stereotypes is selected and applied. Similar to the procedure of applying stereotypes, in UnapplyStereotypeMutation, we search for all features which have applied stereotypes, and unapply a random number of stereotypes.

To sum up, we wrote a framework documentation not only for our purpose but also for further developers who are working on the framework. The framework description is split up into three parts. The first part, Basic Overview, is for people whose aim is to use the Fuzz Testing Framework. In the second part we describe how a concrete Fuzz test can look like. A more technical description can be found in the third part where we describe the framework in greater detail for future developers who want to extend the framework and, therefore, need a deeper understanding of its architecture and how it should be extended.

Furthermore, we extended the EMF Fuzz Testing Framework for generating UML models.

After being able to generate UML models using the generic framework, we finally implemented the profile mutations for randomly applying and unapplying both profiles and stereotypes.        

For more detailed information on our project, we kindly refer the interested reader to our full report and the source code of the developed mutations.