Sunday, April 22, 2012

Unit Testing with CppUnit and Eclipse

As I already mentioned in one of my previous posts, CppUnit is a powerful framework that allows us to automatize the unit testing execution.Unit testing is an extremely important activity in our daily development life, not only from a theoretical quality-related point of view but also from a practical perspective like ensuring the behavior of our code before refactoring, preventing fixed bugs to appear once again etc.

The idea behind CppUnit is to develop parallel classes that will be the ones in charge of testing our 'real' classes. But let's have a look with an example.

1. Installing CppUnit

The first step of course is to get the tool and install it. You can download the last version from sourceforge (at this time 1.12.1) :

http://sourceforge.net/projects/cppunit/files/cppunit/

Untar the downloaded file, and from a terminal move inside the resultant directory and execute the following commands:



./configure
make
sudo make install




2. Using CppUnit for testing our Shared Libraries


A. Creating a shared library to be tested


Now that CppUnit is installed in our system it is time to use it. As example, I will test a shared library that under Eclipse. You can find how to create shared libraries on this previous post:

http://linuxtortures.blogspot.fr/2012/03/shared-libraries-with-eclipse-on-8664.html

Basically open eclipse and click on File --> new, and under the opened dialog select Shared Library, and Linux GCC



As we are under a 64 bits architecture, do not forget to compile with -fPIC, otherwise you will get the following error:

/usr/bin/ld: ./Calculator.o: relocation R_X86_64_32S against `vtable for Calculator' can not be used when making a shared object; recompile with -fPIC

You can activate -fPIC compilation under eclipse by right click on your project and under properties check the following option:





Our library will contain a class 'Calculator' that implements two methods: add and sub (please note that both method are inlined for the shake of simplicity):



A. Creating the testing project

Now let's create our test project, that it will be just an an executable linked to our shared library and of course to the CppUnit framework.

First create a new executable project:





Now link your project against libdl, libcppunit and of course the result of the shared library that we are planning to test. You will need to include the folder where the shared library is generated on the library search path:


The libdl is needed in order to avoid errors of the kind:

/usr/local/lib/libcppunit.so: undefined reference to `dlsym'
/usr/local/lib/libcppunit.so: undefined reference to `dlopen'
/usr/local/lib/libcppunit.so: undefined reference to `dlclose'

You will also need to include the path to our shared library headers in order to use it:




B. Creating the testing class:

Once finished ((almost)) all the including and configuring steps let's focus now on the 'pure' testing procedure.

The most basic case is to just create a class that inherit from CppUnit::TestCase and perform all the needed 'ASSERTS' on the overridden method 'runTest()', then on the main method of our testing project we just need to create a 'CppUnit::TextUi::TestRunner'  and include our just created class as a test to be run:

Here below you can find the code for our test class:



And here the one corresponding to the main method (the one actually in charge of of execute the tests):



That's all, the last step you have to perform before running your test project is just to include your shared library on the environment of the run configuration:


Now you can run your testing project an have a look to the results on the console:





2. Making things more sophisticated by using TestFixture and TestSuite


What we have explained until now is quite nice and it should allows you to start writing your unit test within a cpp environment, but let's complicate the things a bit more and let's talk about testsuites and fixtures.

As you can see on the previous example, we are using the class TestCalculator and its overridden method runTest to create a test. This approach could be enough for most of the cases, but in some cases trying to test the functionality of a library by writing all the asserts in one method or by creating a lot of classes (one per test)  can lead to confusion.

Here is where the TestFixture and TestSuite suites arrive. A TestFixture, basically allows us to convert methods of a class into a testcase, and a TestSuite will allows us to group all those tests and execute them as if it were just one. But let's see all that with an example:

A. Modifying TestCalculator

Now our TestCalculator.h class is going to look like that:


#ifndef TESTCALCULATOR_H_
#define TESTCALCULATOR_H_


#include <Calculator.h>
#include <cppunit/TestFixture.h>
#include <cppunit/TestAssert.h>
#include <cppunit/TestCaller.h>
#include <cppunit/TestSuite.h>


class TestCalculator:  public CppUnit::TestFixture {
public:
TestCalculator();
virtual ~TestCalculator();


 static CppUnit::Test *suite()
 {
   CppUnit::TestSuite *suiteOfTests = new CppUnit::TestSuite( "CalculatorTestSuite" );


   suiteOfTests->addTest( new CppUnit::TestCaller<TestCalculator>(
                                  "Test Sum",
                                  &TestCalculator::testSum ) );
   suiteOfTests->addTest( new CppUnit::TestCaller<TestCalculator>(
                                  "Test Subtraction",
                                  &TestCalculator::testSub ) );
   return suiteOfTests;
 }


 void setUp()
 {
 }
 void tearDown()
 {
 }


private:
 Calculator calculatorToBeTested;


protected:
  void testSum()
  {
 CPPUNIT_ASSERT(calculatorToBeTested.add(1,2)==3);
  }
  void testSub()
  {
 CPPUNIT_ASSERT(calculatorToBeTested.sub(5,1)==4);
  }


};


#endif /* TESTCALCULATOR_H_ */


As you can appreciate there is a static method returning a class Test that indeed returns a TestSuite. This is can be easily understood by having a look to the following diagram:


Indeed it is a composite pattern (Design Pattern) so basically a TestSuite has the same type than TestCase and on the top of everything can indeed contains TestCases.

On the other hand, test caller also inherit from TestCase so with this statemen:


    suiteOfTests->addTest( new CppUnit::TestCaller<TestCalculator>(
                                   "Test Sum",
                                   &TestCalculator::testSum ) );


We area adding TestCases to our TestSuite on the form of a methods (testSum and testSub). It is also worth to mention that setUp method is called each time before executing the actual testMethod and tearDown right after executing it, allowing us to set some pre-conditions (in case of needed) release them.

The last modification we have to perform to have everything working is on our main method:


#include <iostream>
#include <cppunit/TestSuite.h>
#include <cppunit/ui/text/TestRunner.h>
#include "TestCalculator.h"
using namespace std;


int main() {


CppUnit::TextUi::TestRunner runner;
runner.addTest(TestCalculator::suite());
runner.run();


return 0;
}



We are now adding the suite returned from the static method to our runner. Here the results:


Note that now we are running two tests. In case we deliberately make one test fail:


  void testSum()
  {
 CPPUNIT_ASSERT(calculatorToBeTested.add(1,2)==4);
  }



Here the result, showing clearly the name of the test that failed, the line, and the assert.



Monday, April 9, 2012

Continuous Integration with Jenkins, Mercurial and CppUnit on C++

Hello again!! Today I am going to talk about Jenkins, an open source tool that enables the continuous integration and delivery for our projects. As usual we will explain on this post how to set up a basic configuration and have the tool working into a local environment, if you are looking for a more complex configuration you can download here for free the book 'Jenkins the definitive guide'.

Basically what we are following with the continuous integration is to set up an environment that periodically and automatically check out our code, builds it and pass a set of regression/unit tests.

Jenkins provides us with a set of features that allows that and much more, as automatic mails in case of broken builds, dashboards showing the tests results, CVS/Mercurial and others version control systems integration and on the top of everything a friendly web based interface allowing everybody to check the results and manage the system.

A. Installing Mercurial on Debian Squeeze

1. Download here the last release for debian

2. Add the key:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -


3. Install the following dependencies:

sudo apt-get install daemon


4. Install the package we downloaded on the step number 1:

sudo dpkg -i jenkins_1.454_all.deb


5. Logs & PID. By default Jenkins is executed as a stand-alone server, you can find the logs and the process id on the following files:

/var/log/jenkins/jenkins.log
/var/run/jenkins/jenkins.pid

6. Jenkins is configured by default to listen on the port 8080, if you fulfilled the previous steps, you should be able to see your jenkins interface by typing 'localhost:8080' on your favorite web explorer:



B. Linking Jenkins with our Project across Mercurial


The way Jenkins interacts with our source code is across mercurial, once Jenkins is properly configured it checks out our code, into a temp folder, builds the source code and executes the test that we have previously configured.

First of all you will have to install and configure mercurial on your machine. You can have a look here in order to know how to do it.

1. Putting our code under mercurial:

Since the objective of this post is not learn how to use mercurial, we will create a 'local' repository in our working directory (in my case /home/javier/workspace/workspace64/workspaceSharedLibrary) by executing the following command from the previous folder:

hg init


Now our source code is under mercurial and therefore can be accessed by Jenkins, let's see how.

2. Installing the Mercurial Plugin on Jenkins:

The mercurial plugin is not installed by default. From the initial Jenkins web page, click on 'Manage Jenkins', and afterwards click on 'Manage Plugins'. Under the available tab, you should be able to find 'Jenkins Mercurial plugin'. Check it and click on the button 'Install without restart'.

C. Configuring a new job

Now we have everything we need to configure our first 'job'. In Jenkins world, a 'job' is a set of task (that can be defined on a shell script) that are automatically executed within a configurable period, typically those tasks are related with building and testing.

From the initial page (localhost:8080) click on 'New job'. Give a name to your (in my case 'My_firs_job') select 'Build a free-style software project' and click 'ok', a new page allowing configure your job should be displayed. On the first part of the screen we are configuring the Mercurial repository (the local one) that jenkins must check out:


By scrolling down, we set up the frequency of the job. By setting 'Poll SCM' and putting ***** on the Schedule we are indicating that each minute, jenkins must check if there have been changes on the code and if it is the case it must executes the job. In this case the build is defined on the shell file 'home/javier/SANDBOX/Jenkins/build'. We will see the content of this file on the next section.


Finally, by scrolling down again, we can set the list of emails that must be notified in case that a build is not broken.



D. Create the build file ( /home/javier/SANDBOX/Jenkins/build)

The most basic build file is the one that executes the makefile that eclipse generates automatically. The variable 'WORKSPACE' is set up automatically by jenkins and defines the temporally directory where the code is check out.

#!/bin/bash
echo "*********building************"
echo $WORKSPACE
cd $WORKSPACE/InitialTest/Debug
echo $PWD
make clean
make


E. Configure the mail


In order to allow mail sending we have to link our jenkins platform with a mail server. In this case I will use my gmail acount for this end.

From the initial jenkins page (localhost:8080) click on 'Manage Jenkins' and then on 'Configure System'. At the bottom of the page you can set E-mail Notification parameters:




D. Executing the job


Now everything is configured and you can manually execute your job:

From the initial Jenkins page, select your job and on the left side of the page click on 'Build now'. A new item will be created on the 'Build History':



By clicking on the Build History item, you can enter to the details of the build. The most interesting information can be found on the 'Console Output' :

Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/My_first_job/workspace
[workspace] $ hg showconfig paths.default
[workspace] $ hg pull --rev default
[workspace] $ hg update --clean --rev default
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
[workspace] $ hg log --rev . --template {node}
[workspace] $ hg log --rev 1633c8b2cc6ab2fd1b40a594eb3cb21b1b9277f6
[workspace] $ hg log --template "<changeset node='{node}' author='{author|xmlescape}' rev='{rev}' date='{date}'><msg>{desc|xmlescape}</msg><added>{file_adds|stringify|xmlescape}</added><deleted>{file_dels|stringify|xmlescape}</deleted><files>{files|stringify|xmlescape}</files><parents>{parents}</parents></changeset>\n" --rev default:0 --follow --prune 1633c8b2cc6ab2fd1b40a594eb3cb21b1b9277f6
[workspace] $ /bin/sh -xe /tmp/hudson7862122761400994217.sh
+ /home/javier/SANDBOX/Jenkins/build
*********building************
/var/lib/jenkins/jobs/My_first_job/workspace
/var/lib/jenkins/jobs/My_first_job/workspace/InitialTest/Debug
rm -rf  ./src/InitialTest.o  ./src/InitialTest.d  InitialTest


Building file: ../src/InitialTest.cpp
Invoking: GCC C++ Compiler
g++ -I"/home/javier/workspace/workspace64/workspaceSharedLibrary/MySharedLibrary" -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/InitialTest.d" -MT"src/InitialTest.d" -o "src/InitialTest.o" "../src/InitialTest.cpp"
../src/InitialTest.cpp: In function ‘int main()’:
../src/InitialTest.cpp:19: warning: unused variable ‘value’
Finished building: ../src/InitialTest.cpp


Building target: InitialTest
Invoking: GCC C++ Linker
g++  -o "InitialTest"  ./src/InitialTest.o   -lopencv_core -lopencv_highgui -lopencv_imgproc -lopencv_features2d -lopencv_calib3d
Finished building target: InitialTest
Finished: SUCCESS


E. Setting Green Balls as success indicator:

By default Jenkins indicates a successful build with a blue ball. If you (like me) consider that green color is more appropriated for this end, you just need to install the 'Green Balls' pluging;

From the initial Jenkins page click on 'Manage Jenkins' then on 'Manage Plugins' and under 'Available' tab find the 'Green Balls' plugin. Select it and click on 'Install without restart'. In order to have the plugin working, you will need to clean the cache of your explorer.

F. Unit Tests


As final step we are going to set up a unit test project that will be automatically executed (as the build) each time that someone commit a change in the central repository.

The first thing we need is to install 'CppUnit'. CppUnit is an open-source project that provides us with a set of utilities allowing include statements like 'assert' in our code. You can find more information about CppUnit in the official project page:

http://sourceforge.net/projects/cppunit/

More information and examples can be found on the CppUnit cookbook:

http://cppunit.sourceforge.net/doc/1.8.0/cppunit_cookbook.html

1. Download and Install.

From the project page, download and untar the last version of the project (1.12.1).

From the untar directory, execute the following commands:



./configure
make
sudo make install




2. Create your test project.

Using eclipse, create a new C++ project and add the following libraries:

libcppunit.so
libdl.so

If you do not include the libdl.so library, you will receive the following errors at building time:


/usr/local/lib/libcppunit.so: undefined reference to `dlsym'
/usr/local/lib/libcppunit.so: undefined reference to `dlopen'
/usr/local/lib/libcppunit.so: undefined reference to `dlclose'


Your project settings should look like this:



3. Make a shell executing the test project and transforming the results

I will dedicate a whole post to CppUnit, but by the time being we will consider enough to have a project working and generating a resulting report test (you can find how to do this on the CppUnit cookbook)

Once we have developed our test project we just need to create a new shell script (/home/javier/SANDBOX/Jenkins/tests) that runs it and transforms the result into a new report 'readable' by Jenkins:





 #!/bin/bash
echo "*********building tests project************"
echo $WORKSPACE
cd $WORKSPACE/CppUnitProject/Debug
echo $PWD
make clean
make 
$WORKSPACE/CppUnitProject/Debug/CppUnitProject
xsltproc /home/javier/SANDBOX/Jenkins/transform.xsl $WORKSPACE/CppUnitProject/Debug/cpptestresults.xml > $WORKSPACE/testresults.xml



In the previous script we are just executing the auto-generated (by Eclipse) make file and applying a xslt transformation that generates 'testresults.xml' from 'cpptestresults.xml'. In order to do that you will need the transformation file (in our case /home/javier/SANDBOX/Jenkins/transform.xsl) with the following content:






<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" indent="yes"/>
    <xsl:template match="/">
        <testsuite>
            <xsl:attribute name="errors"><xsl:value-of select="TestRun/Statistics/Errors"/></xsl:attribute>
            <xsl:attribute name="failures">
                <xsl:value-of select="TestRun/Statistics/Failures"/>
            </xsl:attribute>
            <xsl:attribute name="tests">
                <xsl:value-of select="TestRun/Statistics/Tests"/>
            </xsl:attribute>
            <xsl:attribute name="name">from cppunit</xsl:attribute>
            <xsl:apply-templates/>
        </testsuite>
    </xsl:template>
    <xsl:template match="/TestRun/SuccessfulTests/Test">
        <testcase>
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
        </testcase>
    </xsl:template>
    <xsl:template match="/TestRun/FailedTests/FailedTest">
        <testcase>
            <xsl:attribute name="classname" ><xsl:value-of select="substring-before(Name, '::')"/></xsl:attribute>
            <xsl:attribute name="name"><xsl:value-of select="substring-after(Name, '::')"/></xsl:attribute>
            <error>
                <xsl:attribute name="message">
                    <xsl:value-of select=" normalize-space(Message)"/>
                </xsl:attribute>
                <xsl:attribute name="type">
                    <xsl:value-of select="FailureType"/>
                </xsl:attribute>
                <xsl:value-of select="Message"/>
                File:<xsl:value-of select="Location/File"/>
                Line:<xsl:value-of select="Location/Line"/>
            </error>
        </testcase>
    </xsl:template>
    <xsl:template match="text()|@*"/>
</xsl:stylesheet>


4. Setting jenkins to execute the new shell

Once everything is in place we just need to include a new shell in our Job. From the initial Jenkins page click on your existing job and on menu placed on the left, click on configure. On the build section click on 'Add Build Step' and select 'Execute shell'. Set the location of our script (in my case /home/javier/SANDBOX/Jenkins/tests) and save the changes.



Now when building our job and checking the console output you should see that the test unit project is executed:


*********building tests project************
/var/lib/jenkins/jobs/My_first_job/workspace
/var/lib/jenkins/jobs/My_first_job/workspace/CppUnitProject/Debug
rm -rf  ./src/CppUnitProject.o ./src/MyClass.o ./src/MyClassTest.o  ./src/CppUnitProject.d ./src/MyClass.d ./src/MyClassTest.d  CppUnitProject

Building file: ../src/CppUnitProject.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/CppUnitProject.d" -MT"src/CppUnitProject.d" -o "src/CppUnitProject.o" "../src/CppUnitProject.cpp"
Finished building: ../src/CppUnitProject.cpp

Building file: ../src/MyClass.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/MyClass.d" -MT"src/MyClass.d" -o "src/MyClass.o" "../src/MyClass.cpp"
Finished building: ../src/MyClass.cpp

Building file: ../src/MyClassTest.cpp
Invoking: GCC C++ Compiler
g++ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/MyClassTest.d" -MT"src/MyClassTest.d" -o "src/MyClassTest.o" "../src/MyClassTest.cpp"
Finished building: ../src/MyClassTest.cpp

Building target: CppUnitProject
Invoking: GCC C++ Linker
g++  -o "CppUnitProject"  ./src/CppUnitProject.o ./src/MyClass.o ./src/MyClassTest.o   -lcppunit -ldl
Finished building target: CppUnitProject



Resources

http://jenkins-ci.org/
http://schneide.wordpress.com/2008/09/29/using-hudson-for-ccmakecppunit/
http://cppunit.sourceforge.net/doc/1.8.0/cppunit_cookbook.html#cppunit_cookbook
http://www.wakaleo.com/books/jenkins-the-definitive-guide