Wednesday, December 31, 2014

Headless Selenium Testing in Django, or Going Deep into the Matrix

I've gotten to the point in developing WhatNext, my experiment in nihilistic task management, that I'm introducing one new bug for each bug I fix.  Proper software development involves testing, and if you skip that you end up paying for it.  Learning the different tools I'm using was so intense and frustrating that I postponed learning the testing tools long enough to get version 0.001 working first so I could have some positive results for my work.  WhatNext now has slightly more functionality than a Google Doc spreadsheet, so I'm using it every day, but it's time to pay the piper.

What that means is automated testing.  Essentially a sparring partner for your program, a program that exercises your program and tests that it does the right thing in every circumstance you can think of, and that it doesn't do the wrong thing.  (Remember, the difference between an alpha version and a real product is that there is at least one way that the alpha works but there should be no way that the product doesn't work.)  With automated testing, you still introduce one new bug for each bug you fix, but you find out immediately and that makes it much easier to fix, as well as reducing the sinking stomach sensation that comes from finding new bugs in production.
I poked around with Django's automated testing tools when I started on WhatNext months ago, and wrote a few little unit tests, but the gold standard of automated testing is a full suite of tests at every level, and the top level is automated browser-based testing.  That is, the computer clicks around itself just like a user would.  One of the key tools for this is a program called Selenium.  And last night I finally gotten Selenium testing to work.

Before I dive into technical notes intended for myself at a future date, let me explain in English.  I can now click a button, or rather type a short command, and my computer will build a complete new copy of my website from scratch, create a new account, and then run Firefox and use Firefox to log in to the website, type some stuff in and save it, check that it was saved properly, edit that stuff, check that it was edited, and delete that stuff, and check that it's actually gone, and then throw away the copy of the website.  This takes under a minute, and I think it can get faster.  This occurs not in "my computer", but in a virtual computer running as a program within my computer, and since that virtual computer doesn't have a screen or anything, Firefox runs in the dark, invisibly.  Basically, I type a command, an absurd amount of computing occurs out of sight, and then I get either an error message or a little
Ran 2 tests in 46.222s
and it does take a bit of faith (and double-checking) to be sure that anything actually happened in there.

(Side note on that: this setup depends on a special program ("Xvfb", or X virtual frame buffer) that pretends to be sort of a computer monitor, but actually throws away everything Firefox thinks is being displayed.  The frame buffer is thing your computer uses to help draw the screen that you see; nowadays it's so trivial that most people never even hear the term. But according to Wikipedia, the first commercial frame buffer was sold in 1974 as special hardware, for $15,000, and supported a 512x512 resolution in gray only, no color (about 2 Kb, I guess).  Today this Xvfb framebuffer is a free program using a tiny fraction of the 16Gb (that's over 8 million times more) of memory in my computer and it wouldn't even occur to me to pay anything extra for it.)

99% of this testing functionality is programs that other people wrote, and the last 1% is sysadmin glue that other people documented, but even so it took six hours over a few weeks (plus a few hours of stabbing in the dark over the last few months, plus a few decades of managing projects that occasionally included excellent test leads that taught me the basics) to get to my first complete CRUD (create, read, update, delete) test.  Here are things I had to piece together, from many helpful answers on the internet, mostly on Stack Overflow, on top of the guides and documentation:

Running headless Selenium and Firefox

Headless selenium tests in django with xvfbwrapper was the start. sudo apt-get install xvfb firefox  is the only external dependency, and the one I'm going to forget the most.


My early tests just returned failure, because Django runs the LiveServerTestCase in non-debug mode so that it matches production.  Which is fine after you get your tests working, less fine before.  To see what's going on, use the decorator
immediately before the function definition. The most convenient way I found to see error messages was to have the test case grab a copy of the web page just before or after the trouble spot.  This code does that:
debug_dump = open('debug.html', 'w')
Of course the dump file isn't easy to see, since this is happening on a headless test server.  I use nginx as a front-end web server in front of the Django python process, even on the test server, so the solution is to have nginx provide a peek at the directory where debug.html is going to live.  Since I run ./ test whatnext from /home/myprojectuser/myproject/, this line in the nginx site configuration file does the trick:
        location /debug/ {
            alias /home/myprojectuser/myproject/;
Then browse to http://testserver/debug/debug.html and refresh as needed.

Conflicts with auth

I use a Django package to provide Facebook login capability, and it is not happy at all running within LiveServerTestCase.  Once I had debugging working, I could see that.  I removed one line of code that does something I haven't bothered to figure out and the problem went away. I may have broken Facebook login on the site but nobody uses it anyway since nobody uses the site so that's a problem for later (this is how programming is done: fix one bug, introduce another).

Random Strings

I also needed to create random strings in Python; there's a great example in Stack Overflow, which I used to write a utility function.  And a bit of digging provided a way to extend that function to provide random unicode strings.  (Seriously, though, the fact that Django and Python 2 are not pure unicode is like building a house with a random mix of metric and imperial tools.  Absurd.)
And I'm looking forward to building some malicious strings with UTF-8 decoder capability and stress test. Then, I'll have to figure out what the Bobby Tables command looks like for Django.

No comments :

Post a Comment