Selenium, Travis-CI and WebRTC == <&

Philipp Hancke

It’s been a while since I last wrote about testing WebRTC applications. And guess what? There has been some progress. We’re now using Travis-CI for integration tests in combination with Selenium for UI testing.

The usage of Travis-CI has been described quite a while ago by our rtc.io friends. It is now being used by adapter.js which has recently become a dependency of our core modules like getUserMedia and RTCPeerConnection since it shims the differences between browsers and is maintained by people from Google and Mozilla.

What Travis does is run a series of tests on every pull request. Thanks to travis-multirunner this is running the testsuite (located here for adapter.js) in different versions of Chrome and Firefox. If the tests pass and jshint and jscs don’t find any style nits, there is a nice green badge:

tests passing

This makes both contributing code as well as merging pull requests easier. No more doubts whether the merge is going to break things. If it does... add a unit test to prevent this in the future. Running the tests in different browser versions also helps detect unexpected API changes before they hit the main release.

So we have code changes covered by those integration tests and some basic protection against browser vendors breaking our application with updates. Let’s go back to testing a WebRTC application like Talky. The approach we used became more problematic as we moved from the simple 1-1 and full-mesh videochat to the more advanced architecture that Talky now uses:

  • 1-1 videochats are still sent between both peers
  • for Firefox users we continue to use the “full mesh” approach until the multistream support in Firefox is stable enough
  • for Chrome users, we upgrade to a session relayed by our videobridge once a third person joins.

This gets rather complex to test with the old approach of filtering out information from the log files.

So what if there was a tool that lets you automate browsers... maybe you heard of Selenium? You probably did, but if you tried it a couple of years back you had to write your tests in Java. Ugh. Wouldn’t it be much cooler if you could write your tests in the same language that your code is written in? Well, it turns out that the JavaScript bindings are now very usable! Which for us means we can write complex tests in a matter of minutes. Here is what the Peer-to-Peer tests look like:

'use strict';
var test = require('tape');

// allow `npm run selenium <url>` usage
var baseURL = process.argv.length >= 3 ? process.argv[2] : 'https://talky.io';

// https://code.google.com/p/selenium/wiki/WebDriverJs
var seleniumHelpers = require('./selenium-lib');
var webdriver = require('selenium-webdriver');

function doJoin(driver, room) {
  return driver.get(baseURL + '/' + room)
  .then(function () {
    return driver.findElement(webdriver.By.id('join'));
  })
  .then(function (button) {
    return button.click();
  });
}

function testP2P(browserA, browserB, t) {
  var room = 'testing_' + Math.floor(Math.random()*100000);

  var userA = seleniumHelpers.buildDriver(browserA);
  doJoin(userA, room);

  var userB = seleniumHelpers.buildDriver(browserB);
  doJoin(userB, room);

  userA.wait(function () {
    return userA.executeScript('return (function() {' +
      'var sessions = app.xmpp.jingle.sessions;' +
      'var sessionIds = Object.keys(sessions);' +
      'if (sessionIds.length != 2) return false;' +
      'if (sessions[sessionIds[0]].peer.full !== sessions[sessionIds[1]].peer.full) return false;' +
      'return sessions[sessionIds[0]]._connectionState === \'connected\';' +
      '})()');
  }, 30*1000)
  .then(function () {
    t.pass('P2P connected');
    userA.quit();
    userB.quit().then(function () {
      t.end();
    });
  })
  .then(null, function (err) {
    t.fail(err);
    userA.quit();
    userB.quit();
  });
}

test('P2P, Chrome-Chrome', function (t) {
  testP2P('chrome', 'chrome', t);
});

test('P2P, Firefox-Firefox', function (t) {
  testP2P('firefox', 'firefox', t);
});

test('P2P, Chrome-Firefox', function (t) {
  testP2P('chrome', 'firefox', t);
});

test('P2P, Firefox-Chrome', function (t) {
  testP2P('firefox', 'chrome', t);
});

What this test does is rather simple. It creates Selenium drivers (using some special options to get a fake camera, see here, then joins a Talky room by navigating to a URL and clicking a button and then waits for the second client to join. The executeScript call is executed every 500 milliseconds until the condition (which checks if the P2P connection got established) is true.

This is done for all combinations of Chrome and Firefox (and maybe soon Microsoft Edge with the recent addition of a Webdriver). The next step here is obviously adding different versions of each browser to the test matrix. Wait, we have done that before... with travis-multirunner. Could those two be integrated?

Turns out this is possible, but it was a little too complicated for me. So I was glad that Google’s Christoffer Jansson took my initial work and connected the dots in the WebRTC samples repository. As a side effect, at least some of those samples are now covered by Selenium tests as well. So if browser updates break the samples, we don’t have to wait for users to report the problem -- which happened just recently.

And at that point, the lazy developer can just sit back and relax while watching browser windows popping up all over the place: selenium doing all the hard work

You might also enjoy reading:

Blog Archives: