Chris Tan

Instant Answer Regression Tester (2014)

I was fortunate to do some contract work for DuckDuckGo when they needed a way to test their Instant Answers library. Instant Answers are keywords that run a command on DuckDuckGo search engine.

Their Instant Answer platform is open source, so anyone can submit one as long as they’re open to do a bit of coding. Here’s an example of one I made that is triggered by the keyword ducksay.


The Challenge

Figure out a way to detect when Instant Answers break. Write a tester that checks all of them (around 300) in less than 5 minutes.

First approach

The first approach was to look for Javascript errors. Initially I was
tasked with testing the Spice IAs which are written in Javascript.

After visiting an IA page with Phantom.js, a script was injected which added an error handler to send results back to the Node.js driver.

I discovered:

Second approach

The second approach used image comparison. It’s a simple solution that detects most problems. It’s also softer when dealing with certain types of errors.

Here’s a screenshot of it running:


When a change is detected it looks like this:


Here’s the GitHub repo of an older version.