Instant Answer Regression Tester (2014)
Figure out a way to detect when Instant Answers break. Write a tester that checks all of them (around 300) in less than 5 minutes.
- Phantom.js is not the most stable
- Many errors were due to an IA’s reliance on a third party API that had changed
- Other types of errors occurred as well, such as the wrong instant answer being triggered
- This approach wasn’t able to detect visual changes such as broken image links or changes to css.
The second approach used image comparison. It’s a simple solution that detects most problems. It’s also softer when dealing with certain types of errors.
- It runs in 30 seconds when using 4 instances of Chrome on a Macbook Air
- Uses WebDriver to control them
- Injects a script which sends the coordinates of the bottom of the IA.
- ImageMagick uses the coordinates to crop a screenshot of the IA before doing an image comparison
Here’s a screenshot of it running:
When a change is detected it looks like this:
Here’s the GitHub repo of an older version.