I recently treked to the 2009 Chicago Code Camp. It was a decent gathering of local talent, including an interesting presentation by Jim Suchy about TDD and Javascript. I was inspired by this talk to create my own JavaScript testing framework.
There are certainly other JavaScript testing frameworks out there. This is no attempt at competing with them. If you happen to like my work, that's awesome. However, I am writing this as a personal development exercise. So, don't expect too much!
Note: Syntax Highlighting to come. The default BlogEngine.NET code syntax highlighter blows.
Usage
One of the take-home points of the presentation (and TDD) was that usage of your code should drive its design. So, I followed format laid out by Jim by setting up my tests first, then implementing the functionality necessary to run those tests in my framework.
jstest( {} );
This will be the primary entry point for the testing framework. It's a function call that will accept a object, whos properties are the functions that will execute the tests. Here's an example usage.
jstest({
myTest: function() {
var a = 3;
var b = 5;
if(a !== b)
throw new Error("a and b not equal");
}
});
The example test will make sure that the variables a and b will be equal. If not, it will throw an exception. The framework should handle this exception and report it. On the framework side, I implemented the functionality that will run this test.
I then added a test that uses an assert object to do the testing.
jstest({
testAssert_FAIL: function(){
var a = 3;
var b = 5;
assert.isEqual(a,b);
},
testAssert_PASS: function(){
var a = 3;
var b = 3;
assert.isEqual(a,b);
}
});
Because I am testing a testing framework, I expect some tests to fail. So, I am marking this with "_FAIL" at the end of the test name. It looks a feels weird, but it's the simplest solution.
The implementation required to support this test adds another global variable to our page's scope, but I hope to keep the count to two. I could have used "jstest.isEqual" or "jstest.assert.isEqual", but neither felt right. I wanted to use the term "assert" because it is used in other testing frameworks. I didn't want to require "jstest.assert" to prepend any assertions because it is long and clunky.
The next leap of test functionality came with adding attributes to the tests. There were a couple of options, here.
jstest({
//test with attributes
myTest1: [
[ /* attributes*/ ],
function(){
//test logic
}
],
//test with no attributes
myTest2: function(){
//test logic
}
});
I considered using a JSON object to declare the tests, but it felt too clunky and unreadable. My next thought was an array, where the first item is a list of attributes and the second item is the function that executes the test. I like using the array notation because it looks like C# method attributes anyway. With a little formatting, it will be clear that these are attributes of the test. In the above example, the tests are allowed to include attributes (by declaring an array) or not (by declaring a function).
I could have used a single array [attr, attr, ..., function(){} ] instead of an array of an array and a function [ [] , function(){} ], but that would 1) messier to implement and 2) less attractive to read on the tests. So, I decided to use the format of the above example.
The implementation of this was interesting. I first focused on the attributes testSetup, testTeardown, suiteSetup, and suiteTeardown. I wrote some tests that used these attributes first to see them fail. Then, I went back to implement the functionality. (I'll get to the code later.)
I had another cycle involving the use of the Ignore attribute. This will be handy for tests that I don't want to run all the time; or, for tests that I have not finished writing.
Once I had passing tests for my other attributes, I decided to add ExpectsException. I wanted an attribute that allowed a test to pass when a specific type of exception was thrown. I also wanted a typed-version, like [ExpectsException(MyExceptionType)], instead of a string-based version, like [ExpectsException("MyExceptionType")]. So, I created a couple tests for this attribute and went down the long path of implementation. (I think that the point of TDD is to use short cycles, but this is a big feature and I'm new to TDD. So, leave your criticisms below.)
The Code!
Let's get to the library code!
//Test constructor
function Test (method, name, attributes) {
this.method = method;
this.name = name;
this.attributes = attributes;
this.expectedExceptions = [];
this.expectsException = function(expected){
for(var ex in this.expectedExceptions) {
if (expected === this.expectedExceptions[ex])
return true;
}
return false;
};
this.expectsAnyException = function() {
return this.expectedExceptions.length > 0;
};
this.expectException = function(ex){
if (ex === null) return;
if (typeof ex !== "function") return;
this.expectedExceptions.push(ex);
};
this.getExpectedExceptionList = function () {
var list = [];
for (var i in this.expectedExceptions) {
//Create a new instance of the exception
// so that we can grab the name
var ex = new this.expectedExceptions[i]();
list.push(ex.name);
}
return list;
};
}
The Test constructor creates a new Test object that has a couple methods to help manage Expected Exceptions. All other attributes are stored in the attributes array for later use. I used an array for expecetedExceptions so that the user can mark more than one. In retrospect, this might not be a great idea. It's probably an indication of a poor test if you expect multiple exceptions. I'll leave it in for now, but it is something that I might remove later.
//Normalizes our tests into our Test type
function normalizeTests(tests){
var normalized = [];
for(var test in tests) {
var name = test;
if(tests[test].constructor === Function) {
normalized.push(
new Test(tests[test], name, null)
);
} else if (tests[test].constructor === Array) {
var attributes = tests[test][0];
var method = tests[test][1];
//Sort attributes by number, then by
// non-number types
attributes.sort(function(a,b) {
if(typeof a !== 'number')
return 1;
if(typeof b !== 'number')
return -1;
return (a - b);
});
normalized.push(
new Test(method,name,attributes)
);
}
}
return normalized;
}
The next function normalizes our test collection. Remember that the user can delcare tests as an array (including attributes) or as a simple function (with no attributes). So, I loop through each item, test its type, and add it to my normalized collection in the appropriate manner. Note that attributes are sorted based on their values, which are defined as integers for simple attributes (Ignore, testSetup, testTeardown, suiteSetup, suiteTeardown) and a function for the ExpectsException attribute. The items are sorted in ascending order for integers and from integers to non-integers. This always gives us Ignore first, if it exists. This is important for the next function, which will setup our test suite.
//Suite constructor
function Suite(tests) {
var empty = function(){};
this.suiteSetup = empty;
this.suiteTeardown = empty;
this.testSetup = empty;
this.testTeardown = empty;
this.tests = [];
var normalizedTests = normalizeTests(tests);
this.removeTest = function (remTest) {
var newTests = [];
for (var test in this.tests) {
if (this.tests[test] !== remTest) {
newTests.push(this.tests[test]);
}
}
this.tests = newTests;
};
for( var test in normalizedTests ) {
var nTest = normalizedTests[test];
if(nTest.attributes !== null) {
for(var a in nTest.attributes) {
var attr = nTest.attributes[a];
if(attr === jstest.Ignore) {
//"Ignore" will be first; so breaking out
//here will skip the other attributes
break;
} else if (attr === jstest.TestSetup) {
if (this.testSetup !== empty)
throw new Error("Cannot use more than one TestSetup attribute.");
this.testSetup = nTest.method;
} else if (attr === jstest.TestTeardown) {
if (this.testTeardown !== empty)
throw new Error("Cannot use more than one TestTeardown attribute.");
this.testTeardown = nTest.method;
} else if (attr === jstest.SuiteSetup) {
if (this.suiteSetup !== empty)
throw new Error("Cannot use more than one SuiteSetup attribute.");
this.suiteSetup = nTest.method;
} else if (attr === jstest.SuiteTeardown) {
if (this.suiteTeardown !== empty)
throw new Error("Cannot use more than one SuiteTeardown attribute.");
this.suiteTeardown = nTest.method;
} else if (attr.constructor === ExpectedException) {
//TODO: Clean this up
this.removeTest(nTest);
nTest.expectException(attr.exConstructor);
this.tests.push(nTest);
} else {
this.tests.push(nTest);
}
}
} else {
this.tests.push(nTest);
}
}
}
This function has a couple of oddities. I want to refactor it to be a little simpler, but it works for now. First, I setup some empty functions for our special test methods. That way, if I call them and they have not been otherwise set, nothing bad will happen.
Then, I declare a helper function to remove a test from the suite. It accomplishes this by looping through each test and pushing it into a new array, ignoring any items that are equal to the test I want to remove. I found some people who use array splicing for this, but I heard about some browser compatibility issues with that method. I want to check on that later, as it would probably be faster. For now, as I said before, it works.
The next step is to loop through all of the tests and then all of their attributes. This is where the ordered attributes becomes necessary. If the Ignore attribute exists in the array, it will be the first item. So, when I test for the Ignore attribute, I can simply break out of the for-loop and be on my merry way.
The other simple attributes work as expected. If I see the attribute, I assign that test to the special placeholder that was defined earlier. In these cases, I don't add the test to the tests collection, because it's not actually a test. When the ExpectsException attribute is detected, I had to do some magic to allow for multiple expected exceptions: I remove the test from the tests collection (if it already exists there), add the expected exception to the test for reference, then add the test to the tests collection. This is another point that I'd like to refactor.
Otherwise, I add the test to the tests collection. This completes the test suite, which now allows me to run a function before the whole suite is run, before each test is run, after each test is run, and after the whole suite has run. It also allows me to expect certain exceptions and ignore certain tests. It's all wrapped up in a nice logical bundle.
Next up is the function that will actually execute the test suite.
//Runs the test suite that we've normalized
function runSuite(suite) {
var passed = 0;
var failed = 0;
function pass(name) {
var msg = '';
addTestResult(name, msg, true);
passed++;
};
function fail(name, exMessage) {
addTestResult(name, exMessage, false);
failed++;
};
suite.suiteSetup();
for(test in suite.tests) {
var sTest = suite.tests[test];
suite.testSetup();
try {
sTest.method();
if(sTest.expectsAnyException()) {
var exceptions = sTest.getExpectedExceptionList();
var exList = exceptions.join(', ');
var msg = 'Expected Exception(s): ' + exList;
fail(sTest.name, msg);
} else {
pass(sTest.name);
}
} catch (ex) {
if(sTest.expectsException(ex.constructor)) {
pass(sTest.name);
} else {
fail(sTest.name, ex.message);
}
}
suite.testTeardown();
}
suite.suiteTeardown();
updateStats(passed, failed);
}
First, I create a couple helper functions to pass and fail the tests. They have access to the pass and fail counters due to the closure that is created, which is awesome.
The logic here is pretty straight forward, until we get to ExpectsException: I run the suiteSetup, foreach test (testSetup, test, testTeardown), and suiteTeardown. Then, I update the stats footer with the number passed and failed.
The messy part comes with expecting exceptions. It's a possible inversion of meaning for the pass/fail-condition of a test. First, I run the test. Then, I check to see if any exceptions are expected. If they are, we would have noticed by now and hit the catch block. So, I fail the test. If no exceptions are expected, I pass the test because we can't have encountered an exception at this point in the code. If there is an exception, I have to check to see if that exception was expected. If so, the test passes. If not, the test fails.
The exception types are tracked by their constructor. There are several built-in types like Error and RangeError, but you can easily make your own. The easiest way to do this is to write a error constructor like the following:
//Custom error type
var MyError = function(msg){
this.message = msg;
this.name = "MyError";
};
throw new MyError("my custom error message");
Then, in a catch block, this exception type can be caught and compared against our ExpectedException types. This is done in the Test constructor with the line "if( sTest.expectsException(ex.constructor) )".
The final pieces of the puzzle are the function that actually accepts the tests and wires the framework together:
var jstest = window.jstest = function(tests) {
jstest.container = createContainer();
var suite = new Suite(tests);
runSuite(suite);
}
...and the attribute declarations:
//Accepts an exception constructor function and
// stores it for later comparison
var ExpectedException = function(exConstructor){
this.exConstructor = exConstructor;
};
//Test Atributes
jstest.Ignore = -1; //this must be the lowest number value of all types
jstest.TestSetup = 1;
jstest.TestTeardown = 2;
jstest.SuiteSetup = 3;
jstest.SutieTeardown = 4;
jstest.ExpectsException = function(ex){ return new ExpectedException(ex); };
Assertions
Assertions are pretty simple. I crafted a simple object that offers the standard set of methods that users can use to test their code. This includes isEqual, notEqual, isNull, notNull, isTrue, and isFalse. (I considered notTrue, but that just sounds silly.) I also created a new Exception type called AssertFailure, which I throw when an assertion fails.
//Assert Exception type constructor
var AssertFailure = function(msg) {
this.name = "AssertFailure";
this.message = msg;
};
//Assert object intializer
var assert = window.assert = {
/*
For isEqual and notEqual, we only show types
if they are 1) different or 2) a primitave type.
Otherwise, we will get things like (object)[object Object],
which doesn't give us any extra information.
*/
isEqual: function(actualValue, expectedValue) {
if (actualValue !== expectedValue) {
var expectedType = typeof expectedValue;
var actualType = typeof actualValue;
if (actualType === expectedType) {
expectedType = actualType = '';
} else {
if (expectedType === 'object') {
expectedType = '';
} else if (expectedType === 'function') {
expectedType = '';
} else {
expectedType = '(' + expectedType + ')';
}
if (actualType === 'object') {
actualType = '';
} else if (actualType === 'function') {
actualType = '';
} else {
actualType = '(' + actualType + ')';
}
}
var msg = "Expected "
+ expectedType + expectedValue
+ " but received "
+ actualType + actualValue + '.';
throw new AssertFailure(msg);
}
},
notEqual: function(valueA, valueB) {
if (valueA === valueB) {
var typeA = typeof valueA;
var typeB = typeof valueB;
if (typeA === typeB) {
typeA = typeB = '';
} else {
if (typeA === 'object') {
typeA = '';
} else if (typeA === 'function') {
typeA = '';
} else {
typeA = '(' + typeA + ')';
}
if (typeB === 'object') {
typeB = '';
} else if (typeB === 'function') {
typeB = '';
} else {
typeB = '(' + typeB + ')';
}
}
//TODO: Update this message to make more sense.
var msg = "Values should not be equal: "
+ typeA + valueA
+ " and "
+ typeB + valueB + '.';
throw new AssertFailure(msg);
}
},
notNull: function(value) {
if (value === null)
throw new AssertFailure("Unexpected null Value.");
},
isNull: function(value) {
if (value !== null)
throw new AssertFailure("Expected null Value, but got " + value + ".");
},
isTrue: function(flag) {
if (flag !== true)
throw new AssertFailure("Expected true.");
},
isFalse: function(flag) {
if (flag !== false)
throw new AssertFailure("Expected false.");
}
};
The "equal" functions are a little clunky because I wanted to show the object type for certain objects and not for others. Otherwise, everything is pretty simple here.
Let's Use It
I have some example tests that I wrote during the process. Many of them will fail, which is good. I refuse to slap an ExpectsException(AssertFailure) on there. I actually didn't expose it outside of my framework, which makes that break. (The framework itself has no dependency on jQuery. I use it below to give me an easy document.ready event handler.)
$(document).ready(function(){
//Custom error type
var MyError = function(msg){
this.message = msg;
this.name = "MyError";
};
jstest({
testAssertEquals: function(){
assert.isEqual(1,function(a,b){return a+b;});
},
test2: function(){
throw new Error("custom error message that is very long custom error message that is very long custom error message that is very long");
},
testAssertNotNull: function(){
var div = document.createElement('div');
assert.notNull(div);
},
testAttributeTestSetup: [
[jstest.TestSetup],
function(){
console.log("Attribute Test - TestSetup!");
}
],
testAttributeTestTeardown: [
[jstest.TestTeardown],
function(){
console.log("Attribute Test - TestTeardown!");
}
],
testAttributeSuiteSetup: [
[jstest.SuiteSetup],
function(){
console.log("Attribute Test - SuiteSetup!");
}
],
testAttributeSuiteTeardown: [
[jstest.SuiteTeardown],
function(){
console.log("Attribute Test - SuiteTeardown!");
}
],
testAttributeIgnore: [
[jstest.Ignore],
function(){
var msg = "DO NOT DISPLAY THIS MESSAGE!";
console.log(msg);
throw new Error(msg);
}
],
//testing ExpectsException
testAttributeExpectedException1_FAIL: [
[jstest.ExpectsException(MyError)],
function(){
throw new Error("testing ExpectedException!");
}
],
testAttributeExpectedException2_FAIL: [
[jstest.ExpectsException(MyError)
,jstest.ExpectsException(RangeError)],
function(){
throw new Error("testing ExpectedException!");
}
],
testAttributeExpectedException1_PASS: [
[jstest.ExpectsException(MyError)
,jstest.ExpectsException(RangeError)],
function(){
throw new RangeError("testing ExpectedException!");
}
],
testAttributeExpectedException2_PASS: [
[jstest.ExpectsException(MyError)
,jstest.ExpectsException(RangeError)],
function(){
throw new MyError("testing ExpectedException!");
}
],
//testing assert.notEqual
testNotEqual_PASS: function() {
assert.notEqual(1,3);
},
testNotEqual_FAIL: [
[jstest.ExpectsException(Error)],
function() {
assert.notEqual(1,1);
}
],
testExpectsException_FAIL: [
[jstest.ExpectsException(MyError)
,jstest.ExpectsException(RangeError)],
function() {
//do nothing
}
],
testNothing: function() {
//do nothing
}
});
});
These tests should give you an idea of the features available in this framework. I'm still weary of allowing multiple ExpectsException attributes. I can't think of a good reason to leave it in the framework. It will probably be left out of the next version.
Download It!
I will probably release this under an open source license, but I want to investigate my options. For now, feel free to download, use, and modify the code. Give me credit for my work and we can play well together. Let me know if you have any suggestions or if you expanded on what I've done here.
jstest.zip (4.56 kb)
On Git
I would like to thank Jim for his interesting presentation.
javascript
javascript, tdd, chicago-code-camp