Wednesday, December 31, 2014

Headless Selenium Testing in Django, or Going Deep into the Matrix

I've gotten to the point in developing WhatNext, my experiment in nihilistic task management, that I'm introducing one new bug for each bug I fix.  Proper software development involves testing, and if you skip that you end up paying for it.  Learning the different tools I'm using was so intense and frustrating that I postponed learning the testing tools long enough to get version 0.001 working first so I could have some positive results for my work.  WhatNext now has slightly more functionality than a Google Doc spreadsheet, so I'm using it every day, but it's time to pay the piper.

What that means is automated testing.  Essentially a sparring partner for your program, a program that exercises your program and tests that it does the right thing in every circumstance you can think of, and that it doesn't do the wrong thing.  (Remember, the difference between an alpha version and a real product is that there is at least one way that the alpha works but there should be no way that the product doesn't work.)  With automated testing, you still introduce one new bug for each bug you fix, but you find out immediately and that makes it much easier to fix, as well as reducing the sinking stomach sensation that comes from finding new bugs in production.
I poked around with Django's automated testing tools when I started on WhatNext months ago, and wrote a few little unit tests, but the gold standard of automated testing is a full suite of tests at every level, and the top level is automated browser-based testing.  That is, the computer clicks around itself just like a user would.  One of the key tools for this is a program called Selenium.  And last night I finally gotten Selenium testing to work.

Before I dive into technical notes intended for myself at a future date, let me explain in English.  I can now click a button, or rather type a short command, and my computer will build a complete new copy of my website from scratch, create a new account, and then run Firefox and use Firefox to log in to the website, type some stuff in and save it, check that it was saved properly, edit that stuff, check that it was edited, and delete that stuff, and check that it's actually gone, and then throw away the copy of the website.  This takes under a minute, and I think it can get faster.  This occurs not in "my computer", but in a virtual computer running as a program within my computer, and since that virtual computer doesn't have a screen or anything, Firefox runs in the dark, invisibly.  Basically, I type a command, an absurd amount of computing occurs out of sight, and then I get either an error message or a little
Ran 2 tests in 46.222s
and it does take a bit of faith (and double-checking) to be sure that anything actually happened in there.

(Side note on that: this setup depends on a special program ("Xvfb", or X virtual frame buffer) that pretends to be sort of a computer monitor, but actually throws away everything Firefox thinks is being displayed.  The frame buffer is thing your computer uses to help draw the screen that you see; nowadays it's so trivial that most people never even hear the term. But according to Wikipedia, the first commercial frame buffer was sold in 1974 as special hardware, for $15,000, and supported a 512x512 resolution in gray only, no color (about 2 Kb, I guess).  Today this Xvfb framebuffer is a free program using a tiny fraction of the 16Gb (that's over 8 million times more) of memory in my computer and it wouldn't even occur to me to pay anything extra for it.)

99% of this testing functionality is programs that other people wrote, and the last 1% is sysadmin glue that other people documented, but even so it took six hours over a few weeks (plus a few hours of stabbing in the dark over the last few months, plus a few decades of managing projects that occasionally included excellent test leads that taught me the basics) to get to my first complete CRUD (create, read, update, delete) test.  Here are things I had to piece together, from many helpful answers on the internet, mostly on Stack Overflow, on top of the guides and documentation:

Running headless Selenium and Firefox

Headless selenium tests in django with xvfbwrapper was the start. sudo apt-get install xvfb firefox  is the only external dependency, and the one I'm going to forget the most.


My early tests just returned failure, because Django runs the LiveServerTestCase in non-debug mode so that it matches production.  Which is fine after you get your tests working, less fine before.  To see what's going on, use the decorator
immediately before the function definition. The most convenient way I found to see error messages was to have the test case grab a copy of the web page just before or after the trouble spot.  This code does that:
debug_dump = open('debug.html', 'w')
Of course the dump file isn't easy to see, since this is happening on a headless test server.  I use nginx as a front-end web server in front of the Django python process, even on the test server, so the solution is to have nginx provide a peek at the directory where debug.html is going to live.  Since I run ./ test whatnext from /home/myprojectuser/myproject/, this line in the nginx site configuration file does the trick:
        location /debug/ {
            alias /home/myprojectuser/myproject/;
Then browse to http://testserver/debug/debug.html and refresh as needed.

Conflicts with auth

I use a Django package to provide Facebook login capability, and it is not happy at all running within LiveServerTestCase.  Once I had debugging working, I could see that.  I removed one line of code that does something I haven't bothered to figure out and the problem went away. I may have broken Facebook login on the site but nobody uses it anyway since nobody uses the site so that's a problem for later (this is how programming is done: fix one bug, introduce another).

Random Strings

I also needed to create random strings in Python; there's a great example in Stack Overflow, which I used to write a utility function.  And a bit of digging provided a way to extend that function to provide random unicode strings.  (Seriously, though, the fact that Django and Python 2 are not pure unicode is like building a house with a random mix of metric and imperial tools.  Absurd.)
And I'm looking forward to building some malicious strings with UTF-8 decoder capability and stress test. Then, I'll have to figure out what the Bobby Tables command looks like for Django.

Wednesday, December 24, 2014

I love computers (for at least another five minutes)

In my last post about Whatnext, I wrote that
Sometimes something that is supposed to be very simple turns out to two minutes of work, IF AND ONLY IF you already know [blah blah blah technical thing]
A big problem with using a software platform as a starting point for development is it locks you into its way of doing things, including using all of the platforms it in turn is built on.  However, sometimes it just works fine (or, I suppose, it's possible that I learning something), and something that should take two minutes not only takes only two minutes, but also saves you from something else that could have taken hours.

Example: In Whatnext, you can filter your task list by a word in the task title or notes.  Here's an example, all of my tasks regarding "windows":

Naturally, I wondered what would happen if I used spaces in the filter text.  Computer programming is generally unkind to spaces, because programmers never reached consensus on whether a space is just another letter like X or 8, or if spaces are magical indications of the difference between each us, or at least our words.  So the safe way to transmit spaces inside of URLs is to change them into something that clearly indicates that a space is just another letter like any other.  That something is often %20, for obvious reasons (that was sarcasm). I anticipated a headache around fixing my code to handle spaces properly.  First, I tried filtering by "wash windows" and was mildly stunned that it worked perfectly:
This is thanks to the Django platform. Since I already wrote this code:

    def get_context_data(self, **kwargs):
        context = super(TaskQueryMixin, self).get_context_data(**kwargs)
        state_params=QueryDict(query_string=None, mutable=True)
        state_params.__setitem__('show_closed', self.request.GET.get('show_closed', ''))
        state_params.__setitem__('show_snoozed', self.request.GET.get('show_snoozed', ''))
        state_params.__setitem__('page', self.request.GET.get('page', ''))
        return context

When I added this new code in the middle:

        state_params.__setitem__('name', self.request.GET.get('name', ''))
        state_params.__setitem__('notes', self.request.GET.get('notes', ''))

it all got a free ride thanks to the .urlencode() function at the end, and it changes "wash windows" into wash+windows (+ instead of %20 because reasons, I guess), and then changes it back at the appropriate time, and it all works fine. I'm nervous to search for a + only to have it fail and break the fragile feeling that I picked a decent platform.

UPDATE: + works fine. It gets encoded to %2B, obviously.

Black yarn requires special conditions to knit

Black yarn can only be knit in direct sunlight; anything else is too dark.  Since I live in San Francisco, this means I can only knit black yarn when I have a south-facing window seat on a flight during daylight hours and we are above the cloud layers.

The remote control pouch is done

Forty-two stitches cast on; then about 25 rows stockinette for the back, then a basket-weave pattern from Vogue (nothing fancy, just knits and purls). Note that the definition of right side and wrong side flips at the fold. To form the pouches, I slipped the upcoming stitch, grabbed the left side of the backing stitch (the left part of the “V” shape that one knit stitch makes to the left needle, then slipped the un-knit stitch back to the left needle, then grabbed the right side of the V, then did k3tog or p3tog as appropriate to the pattern. Something similar for the edges. So the whole thing is knit, no sewing.

I threaded a coathanger through the top for stiffness, but it still sags; maybe doubled coathanger or a dowel or something would work better.

Also knitting is probably the wrong art for a pouch since knit fabric is stretchy, but I don’t sew.


Tuesday, December 23, 2014

Monday, December 22, 2014

Helpful advice for securing your computer ststem

This bit of very helpful advice came from the help desk today:
We remind you to follow these guidelines when dealing with these exploits:

  • When receiving and reading email, read all content thoroughly, even if it appears to be from a trusted source. Malicious emails, in particular, often show signs of poor grammar and spelling
The title of the email?
Ststem Security Alert

Saturday, December 20, 2014

The Bose QC25 and its affect on gay marriage

I wrote earlier about my fear
... that Bose will come out with some new model that is directly comparable to what I have but newer, and therefore better, and that this will cause my headphones to be unsatisfactory, even though they will not physically have been altered in any way by the release of a new product.
It finally happened, with the release of the QC25, but I was able to scour the reviews to find this reassurance:
The new model offers roughly the same amount of noise cancellation ... it’s a fairly mild update to the line and only offers a bit of an improvement, so QC15 owners shouldn’t feel left out.
Meanwhile, I was listening to the new TV On The Radio album, and the first song (Quartz) was so good that I couldn't bear to listen to it straight through, but had to go listen to the song it reminded me of, Lilac from High Life by Brian Eno and Karl Hyde.  Even though Quartz is a pop song, from an album noted as a move "more toward accessibility", and therefore by definition as easy to listen to as possible, I wanted to interrupt consuming it in order to consume Lilac again.

This is a freaky impulse.  Related: I wasn't listening on my QC15s, but instead on my Sure SE215 headphones, which are also noise-cancelling but only because they integrate dense foam plugs intended to be shoved in as earplugs, so they are less comfortable but much smaller than the QC15s, and thus much easier to throw into by backpack for a trip.  But the first time I listened to music through in-ear noise-cancelling headphones was fifteen years ago, a different model, and even though that model probably wasn't as good as what I have now, the experience of standing on my balcony overlooking noisy streets and putting in the headphones and experiencing perfectly isolated music was a high that nothing has touched since then.  In exactly the same way, the first time I experienced noise-cancelling magic, in a mall store, has never been equalled by headphones I've actually owned, even though they are certainly better than that mall demo set.

I share these feeble stories of consumer anxiety as introduction to a fresh batch of stories about how the developed world's social/political/economic system is sliding into disaster:

Douglas Rushkoff writes about the limitations of "jobs", the organizing principle of modern economies.

... we are attempting to use the logic of a scarce marketplace to negotiate things that are actually in abundance. What we lack is not employment, but a way of fairly distributing the bounty we have generated through our technologies, and a way of creating meaning in a world that has already produced far too much stuff.

The Telegraph offers a Niemöller for the middle class:

OK, you might say, but this has always been going on. But it hasn’t. This sort of utterly amoral screw-everyone capitalism has become much more prevalent in the last 15 years. Our financial elite is now totally out of control. They learned nothing from the crisis, except that the rest of us were stupid enough to give them a second chance. And, now, having plucked all the “low hanging fruit,” they’re destroying the middle classes for profit. 
... the sun is still shining and we can still (just about) afford the nanny. But for how much longer? The locusts are already well into the middle middle classes – you know, those poor schmucks who make, say, 40K a year.  ...
This warning might be more powerful if the author wasn't inverting Niemöller; rather than expressing solidarity with the poor, the working class, and "middle middle", the author seems merely to want the economic looting by the rich to stop short of affecting him.  Which sort of misses the point of lamenting not speaking out when they came for the Socialists and the Trade Unionists.

The connection between the two essays, in my mind, is that the the entire construct of "jobs" is integral to a system where "they" can "come for" different groups.  If your sole political motivation is to fight for someone to give you a job, you are fighting to preserve a system set up to exploit you.  I don't discount the emotional and social power of jobs, of stability, of valuing one's self through one's contribution to society in exchange for the tools of living.  I conceed that being worried about my different fancy noise-cancelling headphones puts me firmly in "consumer" mode rather than citizen mode.  But I want to know what a post-capitalist system looks like, where having nice things like fancy headphones doesn't require either winning some kind of birth lottery or being willing to take things from other people.  And of course our current system make those of us born to wealth (in global terms) complicit, because by buying nice stuff and traveling and being a consumer and getting an Ivy League education just means that all of the violent taking is being done on our behalf, out of our sight, by a "system" as much as by cartoon villains, whether that's slave-made consumer electronics, or exported pollution, or colleges whose endowments include centuries-old slave profits.

Anyway, the headphones are pretty nice.

Sunday, December 14, 2014

Sunday, December 7, 2014

The DevOps documentation you are looking for

This document is currently being used in production at several large sites, but there are some experiences and intimations which scar too deeply to permit of healing, and leave only such an added sensitiveness that memory reinspires all the original horror.
One of the more aggravating elements of software development is that in involves a lot of administrating of computers.  The newest nickname for this is "DevOps", for Developers doing Operations.  I briefly considered learning Puppet before deciding to postpone for as long as practical the automation of server administration for my projects.  Finally, however, the documentation I needed is available, thanks to this project that combines Puppet documentation with HP Lovecraft.
Then I came to a downward incline and began its descent—though after a time halted by a gaping, ragged chasm whose narrowest point could not be used in a client/server formation
The Doom that Came to Puppet

Thursday, December 4, 2014

Recursive problems

I can't remember the name of the thing that tells you what the names of things closely related to a thing are.

Wednesday, December 3, 2014

Computer designers have failed us yet again

Two google interfaces have been chafing my eyeballs lately, and I just figured out why.  First: When I buy music on Google Play, and I want it on my phone, I can download it fairly easily, but Google intentionally makes it inaccessible to other music programs so that you are forced to use Google Play, and Google Play isn't very pleasant.  Why not?  Because it seems to be oriented around curating my music for me and also trying to sell me more music ("Recommended for you").  Here's the home screen:

What I actually want is a list of albums that I have downloaded to the phone, and a clear list of what it's going to play next.  This is possible, but it takes a lot of fussing to get here and it's still more glossy than functional.  All I need is the artist name, but look at what percentage of screen pixels is dedicated to showing artist names.  Maybe 5%?

The other culprit is Google+, which automatically uploads all of the photos I take from my phone (great) but makes it very hard to actually find them.  When I go to "Photos" in Google+, I see something different every time, it seems.  Today I got a list of photos I've published on my blog.  Where are the uploads?  And instead of a clear directory of photos in something like a file tree, there's a bunch of photos.  I have thousands of photos.  Instead of letting me browse to "Uploads from Phone" and then going by date, I have to stumble through an ever-changing interface from the Google+ interface team.

Today's epiphany is that they are probably designing for people who would rather quit using a computer than navigate a "file tree".  This shiny interface is probably going to show them what they want only 70% of the time, let's say, and the rest of the time they'll fail, but presented with an interface that requires a working mental model, like a file tree, they'll quit 99% of the time.  It's as though, when you went to a mall, the most popular three or four stores were shuffled to be in front of you, and there was no directory.  Of course, at the mall, if people don't see the store they want, they can at least stumble forward until they smell pretzels, but the average person looking for some data on their phone or even computer still can't do the virtual equivalent of stumbling past Hot Topic toward the food court.

And I think that's the real problem and it's more an indictment of the computer industry than of users.  Computer interfaces are so inconsistent and badly designed that the typical user is asked to develop a new mental model each time they try to use a computer.  Navigating a mall is not cognitively trivial: you have to understand floors and elevators and escalators, and food courts, and kiosks, and anchor stores, but these things have been consistent around the world since before most mall users were born, so that's only one model to learn.  But to use a phone effectively you have to know crap like whether, for the specific app for the specific model for the specific operating system you are using, a double-tap or a long press does anything useful.  Since individual designers have failed the collective action problem of coalescing on a consistent style of interface for mobile phones, and nobody's invented a concept of organizing user data that is both powerful and cognitively accessible to the typical user, now they just throw up shiny stuff on the screen and hope they are guessing what we might have wanted to see.  And make sure they are leaving plenty of room for the "Buy more stuff" buttons.

So Much Work In Progress

Such work.  So progress.

Hat #39. Going very slowly because of the size 0 needles, not so much fun.

Hat #40.

Monster booty, ears almost ready.

A mystery.

More at Ravelry.  I can't wait to finish all of these so I can go shopping for more yarn.

Some of the best things I read on the internet in 2014

Here are a few of the best or most influential of the thousands of articles I read on the internet this year:

The Case Against Early Cancer Detection

If a thousand women have annual cancer exams throughout their 40s, more than half of them will get a false positive test at some point, 60-80 will get unnecessary biopsies, about 10 will be "cured" of a cancer that wouldn't have killed them, and perhaps 1 life will be saved.  The case that everybody should get screened all the time is not a clear-cut one.

Gird Gigerenzer's 2002 book about risk enlightened me about how our brains tend to process statistics in a way that amplifies the damage of false positives in medical testing.  I learned that we should be far more skeptical of the benefits of many kinds cancer tests.  One piece of this is that five-year cancer survival rates are misleading for most things we want to use them for.  The relevant questions are, will detecting this cancer early improve my chances of survival, and is testing for this cancer accurate enough to give me useful information?  Gird strongly suggested that these questions were more up for debate than the pink fundraising brigade would have you believe, but my sporadic searches never managed to find the specific data to finish the argument.  A decade later, FiveThirtyEight finally provides some of it in this article.

Racism in America.

These two sentences are probably the most important corrections I would make to the story of American history I learned in school:
Reconstruction was the second phase of the Civil War. It lasted until 1877, when the Confederates won.

The Case for Reparations
In 2005, Wells Fargo promoted a series of Wealth Building Strategies seminars. Dubbing itself “the nation’s leading originator of home loans to ethnic minority customers,” the bank enrolled black public figures in an ostensible effort to educate blacks on building “generational wealth.” But the “wealth building” seminars were a front for wealth theft. In 2010, the Justice Department filed a discrimination suit against Wells Fargo alleging that the bank had shunted blacks into predatory loans regardless of their creditworthiness. This was not magic or coincidence or misfortune. It was racism reifying itself. According to The New York Times, affidavits found loan officers referring to their black customers as “mud people” and to their subprime products as “ghetto loans.”
It hasn't stopped.  The collective wealth of the United States, which mostly accrues to the richest white people, is in large part stolen from African Americans and from Native Americans.

Why Psychologists’ Food Fight Matters

At least 10 of the 27 “important findings” in social psychology were not replicated at all. 

The link between the first two topics is, a big area of modern life in which the conventional wisdom is very firm and demonstrably wrong.  Increased cancer screening is not always or even usually a good idea; racism in the US was not "resolved" by the Civil War or at any time since. Why are these ideas so strong?  I suppose that as an atheist I'm already used to living in a civilization in which things that are obviously false are believed by 50% or more of my fellow humans, but I'm still surprised that this pattern of disagreement on fundamental aspects of reality is so pervasive.  (Ubiquitous violent sexism in computer culture, such as video games and Silicon Valley companies, was another area in which conventional wisdom started to break up in the last few years.)  This article about (not) replicating experimental results is not as well written as other picks.  But it touches on many fascinating flaws of truth-seeking in modern science, our best truth-seeking institution.

The Judgement of Paris

The Judgement of Paris, as I assume you are aware, is one of the most popular and also the best themes in classical European painting, because it’s based on a legend where three supremely powerful goddesses asked a worthless male mortal to rank them in order of attractiveness in order to win a sculpture of a fruit. Which says so much in such a short amount of time about ancient Greek sexual politics, I think; Yes definitely the Queen of Heaven wants to know if some Trojan shepherd thinks she’s still hot.
I finally discovered The Toast this year.  I don't know what was wrong with me before, but it's fixed now.

Want better, smaller government? Hire another million federal bureaucrats.

We have too few federal bureaucrats monitoring too many grants and contracts, and handling too many dollars. ...  progressively fewer federal bureaucrats have been progressively more responsible not for managing government programs, but for managing proxies who manage programs on the government’s behalf.
On how the US government functions or fails to function, an argument for reversing the trend toward contracting out the bulk of federal duties. Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management

An analysis of how Centers for Medicare & Medicaid Services (CMS) botched's development.  Since I do IT development management for the federal government, I'm particularly interested in how the federal government does IT.  If you combine this article with the previous one, the strong suggestion is that having federal employees manage contracts to get IT development has been a disaster. 

I view IT contracting as similar to teaching, in that both teaching and the subject being taught are separate skills, and the most successful teaching requires the teacher to have both.  Managing an IT contract is a skill, involving knowing the regulations, the ways the regulations are abused, the shortcomings of the regulations and how to overcome them, and so forth.  On top of that, a great contract manager needs to know as much about development as the contractors.  And on top of that, they need to be an expert in the subject.  So that's three separate skills.  I haven't had much direct experience with federal IT contract managers, so I don't know myself, but I suspect that the typical federal IT contract manager was promoted from line work in their subject, and is, at best, skilled in only that one of the three areas (subject, IT development, contract management). 

Today's lunch: a "sandwhich"

The message board in the elevator at work advertising the daily lunch special.  Today that included a "sandwhich."  You may spot the obvious error but only a true grammar prescriptivist could spot the real problem: it should be "sandthat".