Open
Description
We should have tests to validate visual output of widgets and apps. These should be runnable both locally and on Travis. We'd want to be able to run the tests for each browser that we support. On CI it might be ok to do only FF, but would be nice to do all.
General options:
- Selenium webdriver is sort of what
flexx.webruntimedoes, but with more control. First time I tried it did not catch FF. Needs exes to be on PATH. Easy support for grabbing screenshots. For chrome it needs a chromedriver. - PhantomJS is a webkit-based headless browser.
- I can make a webkit browser with high control via PyQt.
- On Linux (and maybe also OSX) I can grab a screenshot from any process that I have the pid of using imagemagix. This could be a good quick way to allow testing on browsers locally.
- On Windows I can do the same (via ctypes).
CI options:
- Travis comes with FF installed and xvfb.
- Appveyor probably has IE.
- Chrome?
- Saucelabs is a service where you can test all OS/browser combos (also mobile). Free for OSS. Can be linked to Travis CI. (pip install from repo to get py3k compat). Uses Selenium to remotely run the VM (run from Travis).
- There is also browserstack.com (free for OSS)
- applitools allows visual testing (semi-automated)
Eventually we'd want to test for several browsers in CI using saucelabs. For the short term, we should aim for a solution that can be run locally and also on Travis.
Approach:
- In the main repo we have a directory with .py files that each describe a static app
- These apps are exported to HTML and published via a git repo. The commit is tagged to make it corresponding to the commit of the main repo.
- We run browser tests on these apps, and compare the results with images in yet another repo. If not on Travis, the system will prompt the user if a new app is present or if a comparison fails. The user can then mark the result as good or bad. Once a good image is present for each app, the images-repo is committed, and tagged with the same commit-id.