Testing
Posted on by Idorobots
Testing in programming is pretty important, no doubts about it, too bad that some languages make it difficult to test the code easly. For instance C++ is pretty limited in terms of DbC and unit testing. All it ever supplied to programmers was a little =assert()= macro that stopped the program providing minimal info about the error and the place it was "caught" in. I mean there are exceptions in C++ and they are pretty usefull, but still it's not enough and requires lots of effort in lerning and implementing correctly. One can still say there are libraries such as CppUnit for those purposes, but we all know how it goes - the more I need to do to get some minor advantage the less I'm interrested in it, and unit testing is so underestimated it's not considered an advantage to have a working piece of code anymore, how about that? And speaking of DbC in C++... It is possible... bam (A sound of a thousand readers going "whaaa" this instant.)
//So we have this class that we want to DbCup a little.
class TisWellBeTestin {
int bar; //It has some fields.
char foo;
public:
int fooBar (char foo) { //And a rather simple method.
this->foo = foo;
return bar;
}
};
How it's done? First stop: class invariant.
class TisWellBeTestin {
int bar;
char foo;
//We make this little private method that checks our objects state
//at every entry and exit of all the methods.
void __invariant() {
assert(bar != 0);
assert(foo != 'A');
}
public:
int fooBar (char foo) {
__invariant(); //Like this...
...
__invariant(); //And this.
return bar;
}
};
So far so good, and it even works. We can clearly see that this was easy, but we got ourselves some extra code that does not look any elegant. Let's move on. Next stop: parameter and return value controll:
class TisWellBeTestin {
int bar;
char foo;
void __invariant() { ... }
public:
int fooBar (char foo) {
//Lets test then... First the parameters.
assert(foo != 'A');
__invariant();
...
__invariant();
//Now the return value.
//This one is a little tricky, we need to store the return value temporary.
int tmp = bar; //Not the happiest example, I know. Deal with it.
assert(tmp != 0);
return tmp;
}
};
Et viola, our TisWellBeTestin class could be considered safe now. Ugly, but safe. There is a way to fix the ugliness a little, though...
class TisWellBeTestin {
int bar;
char foo;
void __invariant() { ... }
public:
int foobar (char foo) {
//We need structs. Structs everywhere!
struct Sentry {
int result;
Sentry() { //Ctor checks our preconditions.
assert(foo != 'A');
__invariant(); //And invariants.
}
~Sentry() { //Dtor checks the postconditions.
assert(result != 0);
__invariant(); //Also invariants.
}
} __sentry; //Preconditions are checked here.
...
//Still a little tricky.
__sentry.result = bar;
return bar;
} //Postconditions here.
};
Looks a lot better now, and we used a STRUCT, I thought these were extinct these days. Anyway, this little example could be improved - we could add #ifdef and #endif not to include the testing code in the release version, or think about optimising all this crap. We still need a unittest that will test our class as a single unit, though.
And now the bad part about it. Imagine we want to derive TisWellBeTestin a couple of times, mixins and virtual inheritence involved. That means, we need to check super classes invariants correctly, and handle virtual functions and the mixins, but what about virtual inheritence... Yeah, you're right, it's not gonna happen.
Moving to Java I've got to admit that I don't know of a better way to use DbC in Java, but I'm sure there is a better way so lets skip to unit testing, this will also apply for C++.
"Hey, this is Java, it's got to be awesome!" Not really, let me get a grip on the old JUnit3 (haven't had a nerve to use JUnit4, pardon me please) and let's see.
//Again, our example class.
public class TisWellBeUnitTestin { //Funny, it fits well into Javas naming convention.
private int bar = 23;
private char foo = 'F';
public int fooBar(char foo) {
this.foo = foo;
return bar;
}
}
//////////Some other remote place (which is good, I know... Sheesh.)
import junit.framework.TestCase; //Note: this isn't part of the language it's an extra package.
public class TisWellBeUnitTestinTest extends TestCase {
public void testFooBar() {
TisWellBeTestin test = new TisWellBeTestin();
test.fooBar('4');
assertEquals(23, test.bar); //1
assertEquals('4', test.foo); //2
}
}
Pretty neat, we've got our test case and we didn't have to add anything to the existing code... Except it won't compile. Note that #1 and #2 are both private. We can either make them public again, or generate getters/setters for them. Neither of these two sounds good to me, though the latter is usually accepted as a solution. Not to mention the need for a separate library in the first place. C++ shares those cons too.
How about D? "Hey Kieth, you're one hell of a handsome stalion and a D enthusiast, tell us how it's done!"
in the background: "I bet it kicks ass..."
Well, yes and no but I'm pretty happy with the way D does this kind of sh... stuff:
//As usually we have our test class...
class TisWellBeTestin {
int bar = 23;
char foo = 'F';
int fooBar(char foo) {
this.foo = foo;
return bar;
}
}
And bam:
//Now with contracts and unittest.
class TisWellBeTestin {
private int bar = 23;
private char foo = 'F';
public int fooBar(char foo)
in { //Preconditions.
assert(foo != 'A');
}
out(result) { //Postconditions.
assert(result != 0);
}
body { //Function implementation.
this.foo = foo;
return bar;
}
invariant() { //Class invariant.
assert(bar != 0);
assert(foo != 'A');
}
unittest { //Unittest.
auto test = new TisWellBeTestin();
assert(test.bar == 23);
assert(test.foo == 'F');
test.fooBar('h');
assert(test.foo == 'h');
assert(test.fooBar('H') == 23);
}
}
All we gotta do is compile this with -debug -unittest (as for DMD compiler), or -fdebug -funittest (as for GNU GDC). It all happens automagically, the pre/post conditions in a function are checked at every entry/exit, so is the class invariant. If we compile it with -funittest it'll automaticly add our unittest block to the binary. If we don't use those compiler switches it gets skipped, as simple as that. It's pretty readable aswell. Works with inheritance too! What's not ok with this? Maby it's just me, but the fully DbC version has three times as many lines. Good grief, look at all the extra code we need, this is outrageus! This is insane! This is madness. It's not worthy the extra time you'd say. Maby it's just me, maby Intelligent IDEs can fix this... Ok, I kind of hate this because I use a simple editor, ugotproblmwitdat?
The other thing is the unittest itself is pretty simple, it doesn't even provide any stats whatsoever. Let's fix it with a little code:
/***********************************************************************************
utils.testing
Simple yet functional testing framework for the D programming language.
Copyright (C) 2011 Kajetan Rzepecki
*********************/
module utils.testing;
/***********************************************************************************
UnittestException
Exception thrown in the unittests, it's more convinient than simple asserts.
*********************/
class UnittestException : Exception {
string what;
this(string type, string file, string line, string expected, string got) {
what = "Unittest assertion (\""~type~"\") failure at "~file~"("~line~"): expected = "~expected~", got = "~got~".";
super(what);
}
this(string file, string line) {
what = "Unittest assertion failure at "~file~"("~line~").";
super(what);
}
override string toString() {
return what;
}
}
/***********************************************************************************
TestCase
A test class.
*********************/
scope class TestCase {
string name;
int run = 0;
Exception[] fails;
this(string name = "") {
this.name = name;
}
/***********************************************************************************
~this
This is a scope class, so its instances get destroyed at the scope end.
This summarizes the test.
*********************/
import std.stdio : writeln, write;
~this() {
write("Test case "~name~" ");
if(fails.length) {
writeln("failed.");
writeln("\tassertions: ", run);
writeln("\tassertions passed: ", run - fails.length);
writeln("\tassertions failed: ", fails.length);
writeln("\tFailures:");
foreach(f; fails) {
writeln("\t\t", f);
}
}
else writeln("passed.");
}
/***********************************************************************************
TestCase.failure
An empty assertion, always to be an error.
*********************/
void failure(int line = __LINE__, string file = __FILE__) {
run++;
fails ~= new UnittestException(file, to!string(line));
}
/***********************************************************************************
TestCase.success
*********************/
void success() {
run++;
}
/***********************************************************************************
TestCase.assertion
e.g. test.assertion!("!=")(3, 3.1415);
*********************/
import std.conv : to;
void assertion(string type, T)(T got, T expected, int line = __LINE__, string file = __FILE__) {
run++;
mixin("if(!(got "~type~" expected))"
"fails ~= new UnittestException(type, file, to!string(line),"
"to!string(expected), to!string(got));");
}
/***********************************************************************************
TestCase.assertion
e. g. test.assertion(2 == 3);
*********************/
void assertion(T : bool)(T got, int line = __LINE__, string file = __FILE__) {
run++;
if(!got)
fails ~= new UnittestException("bool", file, to!string(line), "true", "false");
}
}
A brief explanation: We count assertions and we collect the exceptions that normally would be thrown (for displaying convinience) if the assertions fail. I have to note, that this is a scope class, and that means it's allocated on the stack unlike regular D classes, AND that means it gets deleted after we leave current scope so we can put all the stats into the dtor. The TestCase.failure method is en equivalent of an empty assertion, TestCase.success is rather selfexplanatory. The TestCase.assertion is at liest a little cheesy for someone not yet touched by the glorious D. It's a template that takes a string and a type as it's template parameters, the string is the kind of relation we want to test something for (e. g. "!=", or ">"), next we see a mixin, and this is something awesome about D - since string literals are immutable, the expression in the mixin can be evaluated and compiled into the code an compiletime. It fulfills the need for most assertion types from JUnit (such as assertEquals, assertTrue etc) with only 5 lines of code. Me rikey! The second TestCase.assertion implementation is a separate, specialized template (note there is no generic implementation of this template) for more convinient boolean checks.
This is how you use this framework:
//Code taken from a minimal Scheme implementation I'm on and about atm.
import utils.testing : TestCase;
unittest {
scope test = new TestCase("Shell");
auto shell = new Shell();
auto wat = new Cell(Cell.ATOM, "3.14");
shell.set("wat", wat);
test.assertion!("is")(shell.get("wat"), wat);
shell = new Shell(shell);
auto lol = new Cell(Cell.ATOM, "2.71");
shell.set("lol", lol);
test.assertion!("is")(shell.get("lol"), lol);
test.assertion!("is")(shell.get("wat"), wat);
shell = shell.parent;
test.assertion!("is")(shell.get("wat"), wat);
try {
shell.get("lol");
test.failure;
}
catch(UndefinedSymbol) {
test.success;
}
try {
shell.get("omfg");
test.failure;
}
catch(UndefinedSymbol) {
test.success;
}
}
The example outcome (not the actual wink, wink):
Test case Cell failed.
assertions: 6
assertions passed: 4
assertions failed: 2
Failures:
Unittest assertion ("!=") failure at src/interpreter/cell.d(67): expected = (wat 32 54.3 3.14 3.141592), got = (wat 32 54.3 3.14 3.141592).
Unittest assertion ("!=") failure at src/interpreter/cell.d(74): expected = #[compound-procedure], got = #[compound-procedure].
Test case Shell failed.
assertions: 6
assertions passed: 4
assertions failed: 2
Failures:
Unittest assertion ("!=") failure at src/interpreter/shell.d(38): expected = 3.14, got = 3.14.
Unittest assertion failure at src/interpreter/shell.d(53).
The actual outcome:
Test case Cell passed.
Test case Shell passed.
2016-02-20: Adjusted some links & tags.