Testing Sequentializer

January 15, 2010 at 2:07 pm · Filed under technical, testing

I’d like to share with you a neat function we wrote to sequentialize testing tasks.

Most of our tests are not concerned with timing, but we do have a few dozen time tracking scenarios. Each scenario re-enacts a user taking particular actions for particular durations. In addition to viewing webpages, we throw in application switches, the computer going to sleep, and other time tracking influences.

We’re particularly interested in how different time tracking triggers interact with each other, especially since some of the notifications can be delayed up to 5 seconds.

An important requirement is that we be able to write a test that plainly lists the actions and durations to take; such as the following:

• Start viewing a web page; say, http://foo.com/bar
• 30 seconds later notify the “user is idle” observer
• 40 seconds later notify the “user is back” observer”
• 10 seconds later start viewing a new web page

We expect to nicely abstract the test framework stuff that checks the actual behavior against the expected behavior (in this case a 30 seconds visit followed by a 10 second visit to web page http://foo.com/bar).

Algorithm Overview

The Javascript tool at our disposal is

, which waits for a given duration before executing a given expression.

Thus, the core of our sequentializer is turning a list of expressions and durations into a setTimeout expression that itself calls setTimeout with an expression that calls setTimeout… The recursive nature of the algorithm is best viewed through pseudocode:

# input items: list of (action, duration) pairs
# [ {action: some function, duration: seconds}, 
#     ...,
#    {action: some function}
#  ]
# goal is to execute items[idx].action, wait items[idx].duration,
# and then execute items[idx+1].action, wait items[idx+1].duration,
# and then execute items[idx+2].action, .... until all actions
# are executed. 
# The duration field, if present, in the last dictionary in items
# is ignored.
#
# input idx: index into items of next pair to execute
sequentialize( items, idx )
        # if there is an action to execute, execute it
        if idx < items.length
                execute( items[idx].action )
        fi

        # if there are more actions to execute, recurse
        if idx + 1 < items.length
                setTimeout( sequentialize(items, idx+1), items[idx].duration )
        fi

Javascript Solution

When we converted this pseudocode to a Javascript sequentializer function, we added a _create_sequentializer helper method to do testing stuff so that each test case need only specify the items data structure.

Here’s the full Javascript:

/**
 * @param items: list of functions to execute.
 *      has the following form:
 * [
 *  {fn, self, args, interval},
 *  {fn, self, args, interval},
 *  ...
 * ]
 */
sequentialize: function(items, idx) {
        var self = this;
        if (idx < items.length) {
                items[idx].fn.apply(items[idx].self, items[idx].args);
                if (items[idx].interval) {
                        setTimeout(function() {
                                self.sequentialize(items, idx+1);
                        }, items[idx].interval);
                } else if (idx+1 < items.length) {
                        // flexible: in case we have actions we don't want to wait on
                        self.sequentialize(items, idx+1);
                }
        }
},

/**
 * Test framework stuff
 */
_create_sequence: function(testrunner, display_results_callback, site, actions) {
        var self = this;
        
        var expected_durations = [];
        var sequence = [];
        
        _iterate(actions, function(key, action, index) {
                if (action.expected_visit) {
                        expected_durations.push(Math.round(action.interval/1000.0));
                }
                sequence.push({
                        fn: action.fn,
                        self: action.self,
                        args: action.args,
                        interval: action.interval
                });
        });
        // append tests to sequence
        sequence.push({
                fn: self.check_visits,
                self: self,
                args: [testrunner, display_results_callback, site, expected_durations],
                interval: 1
        })
        // initiate sequence execution
        self.sequentialize(sequence, 0);
},

// here is the above example test case in code
// it reenacts a user initiating a view of a webpage, going idle,
// coming back, and then ending that page view
self._create_sequence(
        testrunner,
        display_results_callback,
        site,
        [{
                fn: self.time_tracker.start_recording,
                self: self.time_tracker,
                args: [site.url],
                interval: Math.floor(Math.random()*7000)+2000,
                expected_visit: true
        }, {
                fn: self.idle_back_listener.idle,
                self: self.idle_back_listener,
                args: [],
                interval: Math.floor(Math.random()*7000)+2000
        }, {
                fn: self.idle_back_listener.back,
                self: self.idle_back_listener,
                args: [],
                interval: Math.floor(Math.random()*7000)+2000,
                expected_visit: true
        }, {
                fn: self.time_tracker.stop_recording,
                self: self.time_tracker,
                args: [],
                interval: 0
        }]
);

Proudly ProcrasDonating,
Lucy.

Comments (1)

UnitTests: PASS

August 27, 2009 at 4:32 pm · Filed under technical, testing

FireBug Console output from new unit testing framework

I integrated QUnit into our extension. This led to a neat bug fix in our encapsulation of jQuery and an excellent refactoring of QUnit.

I encountered three problems:

1. namespace clashes
2. jQuery 1.2.6 incompatibility and
3. a bug in our encapsulation of jQuery

The solutions, which I very much enjoyed implementing, are:

1. refactor QUnit into its own namespace
2. separate test execution data and logic from displaying test results
3. fix bug

I’m going to go through each problem and solution in turn.

Namespace classhes

To include QUnit in the extension meant adding QUnit’s testrunner.js file to the extension’s lib directory and loading it in our XUL overlay. (I added it to lib rather than ext because of how much it diverged from the original. For one, it can no longer receive repository updates.)

The original testrunner.js file adds “test” and “module” among other names to the global namespace. I’m not sure if this was clashing with FireUnit, which I had previously tested, or what, but it looked less than ideal:

(function($) {
    function test(name, callback) {
        ...
    };
    ...
})(jQuery);

I decided to make a TestRunner class using the standard procedure we use elsewhere in the extension: define a “constructor” function and then add instance methods to the function’s prototype object.

Now the code looks like this:

var TestRunner = function(request) {
    this.request = request;
    ...
};
TestRunner.prototype = {};
_extend(TestRunner.prototype, {
    // executes a named test, which itself contains assertions to process
        test: function(name, callback) {
            ;...
    },
    ...
});

Which gets used like this:

var testrunner = new TestRunner(request);

// run some tests
testrunner.test("a happy test", function() {
    testrunner.expect(2);
    testrunner.ok( true, "this test is fine" );
    var value = "hello";
    testrunner.equals( "hello", value, "We expect value to be hello" );
});

note: request is an instance of our own PageRequest object, which encapsulates every HTTP request. Each request contains a jQuery property that encapsulates calling jQuery with the right window and whatnot.

jQuery 1.2.6 incompatibility

Object model for new unit testing framework

For reasons that I wish I’d recorded better, jQuery 1.3.2 and FireFox Exntension-land didn’t get on well. This is why we use jQuery 1.2.6.

QUnit displays test results by inserting elements into the DOM. Some of the jQuery calls relied on version 1.3.2 features, such as

appendTo( selector )	Returns: jQuery
Append all of the matched elements to another, specified, set of elements. As of jQuery 1.3.2, returns all of the inserted elements.

At first I tried to make the least possible changes to get the tests to execute and display. This eventually revealed a bug in our request.jQuery call, which I explain in the third part of this post.

Still, the bug wasn’t the whole pictures. Mixing test execution results with display is madness! A “quick” hack was just wasting time [edit: it took a day of work to yield a not-working "hack", and another day of work to yield a working re-factoring].

In the interest of new developer sanity (mine), maintainability and extensibility, I created two new files:

• testobjects.js
• testrunner_display.js

testobjects.js contains classes that represent the aggregation of assertions in QUnit:

• Assertion – The simple “ok” and “equals” assertions inside test()’s callback.
• TestGroup – Created with QUnit’s “test” function. The callback parameter contains assertions to execute. Each callback executes in its own environment.
• TestModule – Created with QUnit’s “module” function. Modules can specify “startup” and “teardown” functions that get executed before and after every TestGroup’s callback is executed.

testrunner_display.js contains an abstract TestRunnerDisplay class, which is extended by TestRunnerConsoleDisplay and TestRunnerDOMDisplay. All subclasses adhere to the same TestRunnerDisplay API, which essentially takes a testrunner instance. To display test results in the QUnit way, feed a testrunner instance to TestRunnerDOMDisplay; to display test results in FireBug’s console, feed that same testrunner instance to TestRunnerConsoleDisplay.

Encapsulation bug

When a user of our extension visits a webpage, our extension creates a PageRequest instance that gets routed through a URL dispatcher to a “view” (in the Django sense).

The request object encapsulates relevant properties such as URL, event, (unsafe)window and jQuery. The jQuery property lets views and templates use jQuery without having to explicitly pass in the right context (Extension-land is complicated).

Originally, it looked like this:

jQuery: function() {
  return jQuery.fn.init(selector, context || this.get_document());
},

Observe that the jQuery referred to inside the function is the external jQuery object created by the jquery library. If we wanted to refer to the jQuery function being defined we’d do

this.jQuery

.

This code was ok for doing things such as

request.jQuery("#highlight").css("background","yellow")

but failed when using other jQuery utilities, such as

request.jQuery.map( location.search.slice(1).split('&'), decodeURIComponent )

testrunner.js uses these utilities, and maybe we want to use them ourselves, too! The solution is to extend the jQuery function, which is an object, with the rest of jQuery’s properties:

jQuery: (function() {
    var jq = function(selector, context) {
    //logger("request.jQuery(): " + selector + this.event);
    return jQuery.fn.init(selector, context || this.get_document());
    };
    jQuery.extend(jq, jQuery);
    return jq;
})(),

[Future Self: We have since created a wide range of tests using this system. ::Pats past self on the back::]

Proudly ProcrasDonating,

Lucy

Comments