Wednesday, December 31, 2014

Headless Selenium Testing in Django, or Going Deep into the Matrix

I've gotten to the point in developing WhatNext, my experiment in nihilistic task management, that I'm introducing one new bug for each bug I fix.  Proper software development involves testing, and if you skip that you end up paying for it.  Learning the different tools I'm using was so intense and frustrating that I postponed learning the testing tools long enough to get version 0.001 working first so I could have some positive results for my work.  WhatNext now has slightly more functionality than a Google Doc spreadsheet, so I'm using it every day, but it's time to pay the piper.

What that means is automated testing.  Essentially a sparring partner for your program, a program that exercises your program and tests that it does the right thing in every circumstance you can think of, and that it doesn't do the wrong thing.  (Remember, the difference between an alpha version and a real product is that there is at least one way that the alpha works but there should be no way that the product doesn't work.)  With automated testing, you still introduce one new bug for each bug you fix, but you find out immediately and that makes it much easier to fix, as well as reducing the sinking stomach sensation that comes from finding new bugs in production.
I poked around with Django's automated testing tools when I started on WhatNext months ago, and wrote a few little unit tests, but the gold standard of automated testing is a full suite of tests at every level, and the top level is automated browser-based testing.  That is, the computer clicks around itself just like a user would.  One of the key tools for this is a program called Selenium.  And last night I finally gotten Selenium testing to work.

Before I dive into technical notes intended for myself at a future date, let me explain in English.  I can now click a button, or rather type a short command, and my computer will build a complete new copy of my website from scratch, create a new account, and then run Firefox and use Firefox to log in to the website, type some stuff in and save it, check that it was saved properly, edit that stuff, check that it was edited, and delete that stuff, and check that it's actually gone, and then throw away the copy of the website.  This takes under a minute, and I think it can get faster.  This occurs not in "my computer", but in a virtual computer running as a program within my computer, and since that virtual computer doesn't have a screen or anything, Firefox runs in the dark, invisibly.  Basically, I type a command, an absurd amount of computing occurs out of sight, and then I get either an error message or a little
Ran 2 tests in 46.222s
and it does take a bit of faith (and double-checking) to be sure that anything actually happened in there.

(Side note on that: this setup depends on a special program ("Xvfb", or X virtual frame buffer) that pretends to be sort of a computer monitor, but actually throws away everything Firefox thinks is being displayed.  The frame buffer is thing your computer uses to help draw the screen that you see; nowadays it's so trivial that most people never even hear the term. But according to Wikipedia, the first commercial frame buffer was sold in 1974 as special hardware, for $15,000, and supported a 512x512 resolution in gray only, no color (about 2 Kb, I guess).  Today this Xvfb framebuffer is a free program using a tiny fraction of the 16Gb (that's over 8 million times more) of memory in my computer and it wouldn't even occur to me to pay anything extra for it.)

99% of this testing functionality is programs that other people wrote, and the last 1% is sysadmin glue that other people documented, but even so it took six hours over a few weeks (plus a few hours of stabbing in the dark over the last few months, plus a few decades of managing projects that occasionally included excellent test leads that taught me the basics) to get to my first complete CRUD (create, read, update, delete) test.  Here are things I had to piece together, from many helpful answers on the internet, mostly on Stack Overflow, on top of the guides and documentation:

Running headless Selenium and Firefox

Headless selenium tests in django with xvfbwrapper was the start. sudo apt-get install xvfb firefox  is the only external dependency, and the one I'm going to forget the most.


My early tests just returned failure, because Django runs the LiveServerTestCase in non-debug mode so that it matches production.  Which is fine after you get your tests working, less fine before.  To see what's going on, use the decorator
immediately before the function definition. The most convenient way I found to see error messages was to have the test case grab a copy of the web page just before or after the trouble spot.  This code does that:
debug_dump = open('debug.html', 'w')
Of course the dump file isn't easy to see, since this is happening on a headless test server.  I use nginx as a front-end web server in front of the Django python process, even on the test server, so the solution is to have nginx provide a peek at the directory where debug.html is going to live.  Since I run ./ test whatnext from /home/myprojectuser/myproject/, this line in the nginx site configuration file does the trick:
        location /debug/ {
            alias /home/myprojectuser/myproject/;
Then browse to http://testserver/debug/debug.html and refresh as needed.

Conflicts with auth

I use a Django package to provide Facebook login capability, and it is not happy at all running within LiveServerTestCase.  Once I had debugging working, I could see that.  I removed one line of code that does something I haven't bothered to figure out and the problem went away. I may have broken Facebook login on the site but nobody uses it anyway since nobody uses the site so that's a problem for later (this is how programming is done: fix one bug, introduce another).

Random Strings

I also needed to create random strings in Python; there's a great example in Stack Overflow, which I used to write a utility function.  And a bit of digging provided a way to extend that function to provide random unicode strings.  (Seriously, though, the fact that Django and Python 2 are not pure unicode is like building a house with a random mix of metric and imperial tools.  Absurd.)
And I'm looking forward to building some malicious strings with UTF-8 decoder capability and stress test. Then, I'll have to figure out what the Bobby Tables command looks like for Django.

Wednesday, December 24, 2014

I love computers (for at least another five minutes)

In my last post about Whatnext, I wrote that
Sometimes something that is supposed to be very simple turns out to two minutes of work, IF AND ONLY IF you already know [blah blah blah technical thing]
A big problem with using a software platform as a starting point for development is it locks you into its way of doing things, including using all of the platforms it in turn is built on.  However, sometimes it just works fine (or, I suppose, it's possible that I learning something), and something that should take two minutes not only takes only two minutes, but also saves you from something else that could have taken hours.

Example: In Whatnext, you can filter your task list by a word in the task title or notes.  Here's an example, all of my tasks regarding "windows":

Naturally, I wondered what would happen if I used spaces in the filter text.  Computer programming is generally unkind to spaces, because programmers never reached consensus on whether a space is just another letter like X or 8, or if spaces are magical indications of the difference between each us, or at least our words.  So the safe way to transmit spaces inside of URLs is to change them into something that clearly indicates that a space is just another letter like any other.  That something is often %20, for obvious reasons (that was sarcasm). I anticipated a headache around fixing my code to handle spaces properly.  First, I tried filtering by "wash windows" and was mildly stunned that it worked perfectly:
This is thanks to the Django platform. Since I already wrote this code:

    def get_context_data(self, **kwargs):
        context = super(TaskQueryMixin, self).get_context_data(**kwargs)
        state_params=QueryDict(query_string=None, mutable=True)
        state_params.__setitem__('show_closed', self.request.GET.get('show_closed', ''))
        state_params.__setitem__('show_snoozed', self.request.GET.get('show_snoozed', ''))
        state_params.__setitem__('page', self.request.GET.get('page', ''))
        return context

When I added this new code in the middle:

        state_params.__setitem__('name', self.request.GET.get('name', ''))
        state_params.__setitem__('notes', self.request.GET.get('notes', ''))

it all got a free ride thanks to the .urlencode() function at the end, and it changes "wash windows" into wash+windows (+ instead of %20 because reasons, I guess), and then changes it back at the appropriate time, and it all works fine. I'm nervous to search for a + only to have it fail and break the fragile feeling that I picked a decent platform.

UPDATE: + works fine. It gets encoded to %2B, obviously.

Black yarn requires special conditions to knit

Black yarn can only be knit in direct sunlight; anything else is too dark.  Since I live in San Francisco, this means I can only knit black yarn when I have a south-facing window seat on a flight during daylight hours and we are above the cloud layers.

The remote control pouch is done

Forty-two stitches cast on; then about 25 rows stockinette for the back, then a basket-weave pattern from Vogue (nothing fancy, just knits and purls). Note that the definition of right side and wrong side flips at the fold. To form the pouches, I slipped the upcoming stitch, grabbed the left side of the backing stitch (the left part of the “V” shape that one knit stitch makes to the left needle, then slipped the un-knit stitch back to the left needle, then grabbed the right side of the V, then did k3tog or p3tog as appropriate to the pattern. Something similar for the edges. So the whole thing is knit, no sewing.

I threaded a coathanger through the top for stiffness, but it still sags; maybe doubled coathanger or a dowel or something would work better.

Also knitting is probably the wrong art for a pouch since knit fabric is stretchy, but I don’t sew.


Tuesday, December 23, 2014

Monday, December 22, 2014

Helpful advice for securing your computer ststem

This bit of very helpful advice came from the help desk today:
We remind you to follow these guidelines when dealing with these exploits:

  • When receiving and reading email, read all content thoroughly, even if it appears to be from a trusted source. Malicious emails, in particular, often show signs of poor grammar and spelling
The title of the email?
Ststem Security Alert

Saturday, December 20, 2014

The Bose QC25 and its affect on gay marriage

I wrote earlier about my fear
... that Bose will come out with some new model that is directly comparable to what I have but newer, and therefore better, and that this will cause my headphones to be unsatisfactory, even though they will not physically have been altered in any way by the release of a new product.
It finally happened, with the release of the QC25, but I was able to scour the reviews to find this reassurance:
The new model offers roughly the same amount of noise cancellation ... it’s a fairly mild update to the line and only offers a bit of an improvement, so QC15 owners shouldn’t feel left out.
Meanwhile, I was listening to the new TV On The Radio album, and the first song (Quartz) was so good that I couldn't bear to listen to it straight through, but had to go listen to the song it reminded me of, Lilac from High Life by Brian Eno and Karl Hyde.  Even though Quartz is a pop song, from an album noted as a move "more toward accessibility", and therefore by definition as easy to listen to as possible, I wanted to interrupt consuming it in order to consume Lilac again.

This is a freaky impulse.  Related: I wasn't listening on my QC15s, but instead on my Sure SE215 headphones, which are also noise-cancelling but only because they integrate dense foam plugs intended to be shoved in as earplugs, so they are less comfortable but much smaller than the QC15s, and thus much easier to throw into by backpack for a trip.  But the first time I listened to music through in-ear noise-cancelling headphones was fifteen years ago, a different model, and even though that model probably wasn't as good as what I have now, the experience of standing on my balcony overlooking noisy streets and putting in the headphones and experiencing perfectly isolated music was a high that nothing has touched since then.  In exactly the same way, the first time I experienced noise-cancelling magic, in a mall store, has never been equalled by headphones I've actually owned, even though they are certainly better than that mall demo set.

I share these feeble stories of consumer anxiety as introduction to a fresh batch of stories about how the developed world's social/political/economic system is sliding into disaster:

Douglas Rushkoff writes about the limitations of "jobs", the organizing principle of modern economies.

... we are attempting to use the logic of a scarce marketplace to negotiate things that are actually in abundance. What we lack is not employment, but a way of fairly distributing the bounty we have generated through our technologies, and a way of creating meaning in a world that has already produced far too much stuff.

The Telegraph offers a Niemöller for the middle class:

OK, you might say, but this has always been going on. But it hasn’t. This sort of utterly amoral screw-everyone capitalism has become much more prevalent in the last 15 years. Our financial elite is now totally out of control. They learned nothing from the crisis, except that the rest of us were stupid enough to give them a second chance. And, now, having plucked all the “low hanging fruit,” they’re destroying the middle classes for profit. 
... the sun is still shining and we can still (just about) afford the nanny. But for how much longer? The locusts are already well into the middle middle classes – you know, those poor schmucks who make, say, 40K a year.  ...
This warning might be more powerful if the author wasn't inverting Niemöller; rather than expressing solidarity with the poor, the working class, and "middle middle", the author seems merely to want the economic looting by the rich to stop short of affecting him.  Which sort of misses the point of lamenting not speaking out when they came for the Socialists and the Trade Unionists.

The connection between the two essays, in my mind, is that the the entire construct of "jobs" is integral to a system where "they" can "come for" different groups.  If your sole political motivation is to fight for someone to give you a job, you are fighting to preserve a system set up to exploit you.  I don't discount the emotional and social power of jobs, of stability, of valuing one's self through one's contribution to society in exchange for the tools of living.  I conceed that being worried about my different fancy noise-cancelling headphones puts me firmly in "consumer" mode rather than citizen mode.  But I want to know what a post-capitalist system looks like, where having nice things like fancy headphones doesn't require either winning some kind of birth lottery or being willing to take things from other people.  And of course our current system make those of us born to wealth (in global terms) complicit, because by buying nice stuff and traveling and being a consumer and getting an Ivy League education just means that all of the violent taking is being done on our behalf, out of our sight, by a "system" as much as by cartoon villains, whether that's slave-made consumer electronics, or exported pollution, or colleges whose endowments include centuries-old slave profits.

Anyway, the headphones are pretty nice.

Sunday, December 14, 2014

Sunday, December 7, 2014

The DevOps documentation you are looking for

This document is currently being used in production at several large sites, but there are some experiences and intimations which scar too deeply to permit of healing, and leave only such an added sensitiveness that memory reinspires all the original horror.
One of the more aggravating elements of software development is that in involves a lot of administrating of computers.  The newest nickname for this is "DevOps", for Developers doing Operations.  I briefly considered learning Puppet before deciding to postpone for as long as practical the automation of server administration for my projects.  Finally, however, the documentation I needed is available, thanks to this project that combines Puppet documentation with HP Lovecraft.
Then I came to a downward incline and began its descent—though after a time halted by a gaping, ragged chasm whose narrowest point could not be used in a client/server formation
The Doom that Came to Puppet

Thursday, December 4, 2014

Recursive problems

I can't remember the name of the thing that tells you what the names of things closely related to a thing are.

Wednesday, December 3, 2014

Computer designers have failed us yet again

Two google interfaces have been chafing my eyeballs lately, and I just figured out why.  First: When I buy music on Google Play, and I want it on my phone, I can download it fairly easily, but Google intentionally makes it inaccessible to other music programs so that you are forced to use Google Play, and Google Play isn't very pleasant.  Why not?  Because it seems to be oriented around curating my music for me and also trying to sell me more music ("Recommended for you").  Here's the home screen:

What I actually want is a list of albums that I have downloaded to the phone, and a clear list of what it's going to play next.  This is possible, but it takes a lot of fussing to get here and it's still more glossy than functional.  All I need is the artist name, but look at what percentage of screen pixels is dedicated to showing artist names.  Maybe 5%?

The other culprit is Google+, which automatically uploads all of the photos I take from my phone (great) but makes it very hard to actually find them.  When I go to "Photos" in Google+, I see something different every time, it seems.  Today I got a list of photos I've published on my blog.  Where are the uploads?  And instead of a clear directory of photos in something like a file tree, there's a bunch of photos.  I have thousands of photos.  Instead of letting me browse to "Uploads from Phone" and then going by date, I have to stumble through an ever-changing interface from the Google+ interface team.

Today's epiphany is that they are probably designing for people who would rather quit using a computer than navigate a "file tree".  This shiny interface is probably going to show them what they want only 70% of the time, let's say, and the rest of the time they'll fail, but presented with an interface that requires a working mental model, like a file tree, they'll quit 99% of the time.  It's as though, when you went to a mall, the most popular three or four stores were shuffled to be in front of you, and there was no directory.  Of course, at the mall, if people don't see the store they want, they can at least stumble forward until they smell pretzels, but the average person looking for some data on their phone or even computer still can't do the virtual equivalent of stumbling past Hot Topic toward the food court.

And I think that's the real problem and it's more an indictment of the computer industry than of users.  Computer interfaces are so inconsistent and badly designed that the typical user is asked to develop a new mental model each time they try to use a computer.  Navigating a mall is not cognitively trivial: you have to understand floors and elevators and escalators, and food courts, and kiosks, and anchor stores, but these things have been consistent around the world since before most mall users were born, so that's only one model to learn.  But to use a phone effectively you have to know crap like whether, for the specific app for the specific model for the specific operating system you are using, a double-tap or a long press does anything useful.  Since individual designers have failed the collective action problem of coalescing on a consistent style of interface for mobile phones, and nobody's invented a concept of organizing user data that is both powerful and cognitively accessible to the typical user, now they just throw up shiny stuff on the screen and hope they are guessing what we might have wanted to see.  And make sure they are leaving plenty of room for the "Buy more stuff" buttons.

So Much Work In Progress

Such work.  So progress.

Hat #39. Going very slowly because of the size 0 needles, not so much fun.

Hat #40.

Monster booty, ears almost ready.

A mystery.

More at Ravelry.  I can't wait to finish all of these so I can go shopping for more yarn.

Some of the best things I read on the internet in 2014

Here are a few of the best or most influential of the thousands of articles I read on the internet this year:

The Case Against Early Cancer Detection

If a thousand women have annual cancer exams throughout their 40s, more than half of them will get a false positive test at some point, 60-80 will get unnecessary biopsies, about 10 will be "cured" of a cancer that wouldn't have killed them, and perhaps 1 life will be saved.  The case that everybody should get screened all the time is not a clear-cut one.

Gird Gigerenzer's 2002 book about risk enlightened me about how our brains tend to process statistics in a way that amplifies the damage of false positives in medical testing.  I learned that we should be far more skeptical of the benefits of many kinds cancer tests.  One piece of this is that five-year cancer survival rates are misleading for most things we want to use them for.  The relevant questions are, will detecting this cancer early improve my chances of survival, and is testing for this cancer accurate enough to give me useful information?  Gird strongly suggested that these questions were more up for debate than the pink fundraising brigade would have you believe, but my sporadic searches never managed to find the specific data to finish the argument.  A decade later, FiveThirtyEight finally provides some of it in this article.

Racism in America.

These two sentences are probably the most important corrections I would make to the story of American history I learned in school:
Reconstruction was the second phase of the Civil War. It lasted until 1877, when the Confederates won.

The Case for Reparations
In 2005, Wells Fargo promoted a series of Wealth Building Strategies seminars. Dubbing itself “the nation’s leading originator of home loans to ethnic minority customers,” the bank enrolled black public figures in an ostensible effort to educate blacks on building “generational wealth.” But the “wealth building” seminars were a front for wealth theft. In 2010, the Justice Department filed a discrimination suit against Wells Fargo alleging that the bank had shunted blacks into predatory loans regardless of their creditworthiness. This was not magic or coincidence or misfortune. It was racism reifying itself. According to The New York Times, affidavits found loan officers referring to their black customers as “mud people” and to their subprime products as “ghetto loans.”
It hasn't stopped.  The collective wealth of the United States, which mostly accrues to the richest white people, is in large part stolen from African Americans and from Native Americans.

Why Psychologists’ Food Fight Matters

At least 10 of the 27 “important findings” in social psychology were not replicated at all. 

The link between the first two topics is, a big area of modern life in which the conventional wisdom is very firm and demonstrably wrong.  Increased cancer screening is not always or even usually a good idea; racism in the US was not "resolved" by the Civil War or at any time since. Why are these ideas so strong?  I suppose that as an atheist I'm already used to living in a civilization in which things that are obviously false are believed by 50% or more of my fellow humans, but I'm still surprised that this pattern of disagreement on fundamental aspects of reality is so pervasive.  (Ubiquitous violent sexism in computer culture, such as video games and Silicon Valley companies, was another area in which conventional wisdom started to break up in the last few years.)  This article about (not) replicating experimental results is not as well written as other picks.  But it touches on many fascinating flaws of truth-seeking in modern science, our best truth-seeking institution.

The Judgement of Paris

The Judgement of Paris, as I assume you are aware, is one of the most popular and also the best themes in classical European painting, because it’s based on a legend where three supremely powerful goddesses asked a worthless male mortal to rank them in order of attractiveness in order to win a sculpture of a fruit. Which says so much in such a short amount of time about ancient Greek sexual politics, I think; Yes definitely the Queen of Heaven wants to know if some Trojan shepherd thinks she’s still hot.
I finally discovered The Toast this year.  I don't know what was wrong with me before, but it's fixed now.

Want better, smaller government? Hire another million federal bureaucrats.

We have too few federal bureaucrats monitoring too many grants and contracts, and handling too many dollars. ...  progressively fewer federal bureaucrats have been progressively more responsible not for managing government programs, but for managing proxies who manage programs on the government’s behalf.
On how the US government functions or fails to function, an argument for reversing the trend toward contracting out the bulk of federal duties. Ineffective Planning and Oversight Practices Underscore the Need for Improved Contract Management

An analysis of how Centers for Medicare & Medicaid Services (CMS) botched's development.  Since I do IT development management for the federal government, I'm particularly interested in how the federal government does IT.  If you combine this article with the previous one, the strong suggestion is that having federal employees manage contracts to get IT development has been a disaster. 

I view IT contracting as similar to teaching, in that both teaching and the subject being taught are separate skills, and the most successful teaching requires the teacher to have both.  Managing an IT contract is a skill, involving knowing the regulations, the ways the regulations are abused, the shortcomings of the regulations and how to overcome them, and so forth.  On top of that, a great contract manager needs to know as much about development as the contractors.  And on top of that, they need to be an expert in the subject.  So that's three separate skills.  I haven't had much direct experience with federal IT contract managers, so I don't know myself, but I suspect that the typical federal IT contract manager was promoted from line work in their subject, and is, at best, skilled in only that one of the three areas (subject, IT development, contract management). 

Today's lunch: a "sandwhich"

The message board in the elevator at work advertising the daily lunch special.  Today that included a "sandwhich."  You may spot the obvious error but only a true grammar prescriptivist could spot the real problem: it should be "sandthat".

Friday, November 28, 2014

TMI Friday

(TMI Alert, as per title)

As much as I love my Swash 1000 (it's Japanese-style), I have to confess that the deeply inadequate blowdry feature means that the environmental savings are less than I'd hoped.

Monday, November 24, 2014

It's hard to be afraid of everything

She touched the foam once with the tip of her nose.  It rocked slightly, and she retreated to a safe distance, where she has been stuck since.

Thursday, November 20, 2014

"Sometimes you're the windshield", or, a rare moment when programming is not all pain

Something worked today. Hold your applause until the end.

I decided to build Whatnext (the nihilistic todo list) with something called Django, which is supposed to handle all of the common website stuff, like logins and tables and sorting.  It mostly does, but it's a very leaky abstraction.  Sometimes something that is supposed to be very simple turns out to two minutes of work, IF AND ONLY IF you already know Python data structures so well that you can instantly describe how each of the following differ:
for i in dir(context): print i
for d in context.dicts: print d.keys()
for i in context["field"].__dict__.items(): print i
If you don't know (I still don't), you'll spend a few hours on that two-minute task.

Well.  Just now I decided I wanted to bite the bullet and learn "messages", which is Django-speak for those nice little announcements that pop up after you do something on a website but don't bother you and go away in a few seconds. Slightly less than seven minutes later, I had this:

How nice is that?  It's about 8 out of 10 nice, I think; it needs to be yellow with drop shadow for 9 of 10 nice, beyond that maybe rounded corners and some spirit fingers?

I'm so relieved and startled to have something be as easy as it was supposed to be.  You're not always the bug; sometimes you're the windshield.

Tuesday, November 18, 2014

Is your task management strategy more Sisyphus or Tantalus?

The premise of WhatNext, my task management program/concept/philosophical exploration, reflects the Johnson quote:
When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.
Given that you have a specific, limited amount of time to do anything before death's warm embrace, which mundane, trivial, or unpleasant thing should you do in the next few minutes and hours of your flash of existence on Earth?

Okay, that's a bit morbid, but lighten up: this blog post is really about whether you approach tasks as Sisyphus, doomed to forever roll a rock up a hill only to have it roll back down, or as Tantalus:
... the gods punished him by placing him in the nether world in the midst of a lake, but rendering it impossible for him to drink when he was thirsty, the water always withdrawing when he stooped. Branches laden with fruit, moreover, hung over his head, but when he stretched out his hand to reach the fruit, the branches withdrew.
I think most people perform Tantalus-style task management.  Do you make lists of things to do?  How does that work?  You go through the list, checking things off, and if you have had a very successful and productive day, you've merely gotten yourself from "behind" back to zero.  That is, the best one can hope for is to not be at a negative.  This is a miserable way to exist.  This mentality is reified in the software development process called Scrum, with the Burndown chart:

In the burndown chart, when all of your work is done, you are at zero.  Congratulations?  You have nothing.  You think you are finished and that you have reached the fruit, but instead of eating the fruit you just start looking for more fruit; it is as if the fruit sprang away from you. In response, some Scrum people have come up with the Burnup chart, where you can see your progress accummulate, and what I use professionally now looks more like this:

On the bright side, you can see all of that lovely green of completed work accumulate.  The other colors represent various stages of work in progress, and the gray the maximum amount of work the teams thought they might achieve in one "Sprint", which is our standard work cycle of four weeks.  On the dark side, however, every 4 weeks we start over:

So while you get to go as far as 28 days before seeing all progress flushed away, eventually the chart does indeed reset to zero, and you are forced to confront the reality that you too are Sisyphus, pushing the rock up over and over again.  Even if you maintain a project burnup that shows cumulative progress over months and years, eventually your project will end, and you will have to find another rock to roll up another hill.

Addressing that is beyond the scope of today's post, although I do think there is something there that Whatnext can help with, not as software per se but as an experiment in a better conceptual framework for getting things done.  As homework for that, however, I want to read this book that Marc Maron kept pushing on his podcast, The Denial of Death.  So far I've read the introduction, and while I really liked it, I can't say that it gave me any immediate ideas for task management software.

To get back to the tangible: Now that I have a minimally working task list program in Whatnext, I want to add the first feature that starts moving it from the Tantalus or Sisyphus model of most task tracking concepts to a more positive model.  If that's not a burndown or burnup chart, what is it?  I want an immediate, positive, visible indicator of progress, so that whenever I complete a task, I can see something that shows that I have spent my time wisely.

A program called RescueTime offers one idea: a "Dashboard" that shows a quantitative moral verdict on your minutes:

In this chart, red is time wasted, and blue is time well spent.  More specifically, blue is "productive" time, and you can numerically judge the quality of my time today by the ratio between the two:

This thread of thought—I beg your pardon, as I started this sentence, RescueTime, as if it was reading what I type (and, given the state of privacy and how little companies respect it in 2014, it may well have been), flashed up this quote:
“The time you enjoy wasting is not wasted time.” — Bertrand Russell
Which is exactly where I was going with this idea.  Before you can judge productivity, you have to categorize all of the ways you can spend time, but even before that, you have to understand what you mean by productivity.  My numbers for today looked much worse just now until I told it that blogging counts "Writing", which is categorized as "Design & Composition", not News & Opinion; that makes it "Very Productive", not "Distracting". What is your product?  Actionable deliverables?  Pages of product?  Clicks?  Revenue?  Acts of kindness?  A sense of satisfaction?  The ability to die in peace and contentment?  What are you being distracted from?  Being productive, or acknowledging the Grim Reaper?  Why does she always have a clock?  I'm very exciting to get into these questions with Whatnext, but I have plenty of philosophical homework to tackle first.

In the meantime, reeling us back from the abyss of nothingness to the simpler hell of management-speak, let's think about Milestones.  Or, let's call them Thresholds.  At the moment, other than blogging, I have a dozen tasks I'm thinking about for Whatnext.  If I arrange them around what they enable me to accomplish, I have a list of new capabilities and achievements instead of a list of tasks to burn through and hide.
  1. "testing".  Instead of a big handful of testing tasks that I've been neglecting, I have the opportunity to save myself from regression bugs in the future.
  2. "Toggl integration".  Instead of a messy API integration that will require me to climb up several brand new learning curves while experiencing the frustrations unique to cross-product integration, I have the opportunity  stop copying and pasting time data between websites.
  3. "Integrating task tracking lists".  Instead of the headache of merging my task lists from four different tools (Redmine, Checkvist, draft emails, Google Spreadsheets), each of which does only a fraction of what I need it to do (which is why I'm working on Whatnext at all), I have the opportunity to .... enjoy freedom from worrying which different list I should be looking at?
  4. "Preparing for a closed Alpha".  Instead of trying to scurry around and fix a billion bugs and catch up on the deeply unfun "DevOps" tasks that I have skipped to date but which are required for a stable, reliable, and safe website, I have the opportunity to share my ideas with some friends.
  5. "implementing thresholds".  Instead of wrestling to articulate a fuzzy concept of a data model and then stab in the dark at how to implement it, all which a severely limited set of tools due to my inexperience at Django, I have the opportunity to simplify how I decide what to do next.
At its worst this is just putting a falsely happy spin on unpleasent or difficult things, but I do think there's a nugget in the concept of Thresholds that is useful and positive.  My idea is that each tag could have some specific milestones (which I'll call thresholds until I get an idea which is both better and actually good), and then all tasks with that tag can be sorted as either needed or not needed for a threshold.  What could make this different from less-that-helpful paper exercises like Gantt charts and rollups and the like is if I can design an interface that makes it simple, fast, and context-specific.  In particular, I think this has to be tied into the idea that there is no two-dimensional wall chart of activities and durations; there is only a one-dimensional list and everything is sequence.  There's only one of you at a time so you can only do one task at a time so all tasks must be sequential.

It might end up something like this:

0) You jot down a bunch of things you think you need to do, and they all jumble in with the dozens of other things you previously jotted down.
1) You pick a tag, i.e., a context, to focus on.  Just stuff for your job, or just household tasks.
2) Within that context, you define one or two outcomes, triumphs, accomplishments.  Things that probably have a lot of separate sub-tasks.  Not tasks, but conditions of success.  "I am ready for my trip next week."  "My home is clear of things that I don't need which can be recycled or donated."  "All of my homework for the rest of the year is completed by captive demon."  Probably you rank or date these somehow to indicate priority.
3) Based on the tags and maybe dates and such, you are able to look at a limited number of tasks, maybe one screen full, maybe that's about 20 or 30, and you move the tasks above and below the thresholds you just defined.  Based on this, the system helps stack-rank all of the tasks.  E.g., "gather all old newspapers into paper bags" is not necessary to get ready for that trip, so if the trip is more important than cleaning out the home, gathering the old newspapers just moved below all trip-related tasks.
4) See at all times in Whatnext some kind of indication of the most important threshold, how far away it is in time: "you have an estimated 2 hours of work to complete to reach this accomplishment and have this result", and what progress is being made: "at your current rate of progress on knitting tasks you will finish those 2 hours of work in 3 weeks".

Where this gets messy: what is the interaction of tasks across different categories and contexts?  How do you stack-rank incommensurable thresholds, such as "get promoted" versus "watch all the good movies"?  What about dependencies?  But all of those things probably belong to thresholds below the five I identified above.  And building a prototypical threshold feature in Whatnext (#5 above) will make it possible for me to figure out what to do next, so all I have to do is achieve threshold #5 in order to figure out that I should probably work on #5.

Meanwhile, she is waiting patiently for you:

Monday, November 17, 2014

Programmer problems: Is "Soft Delete" evil?

This is part of what I guess will be an ongoing series, "There's one right way to do that and it's wrong."  Today it's about deleting things: what's better, a confirmation or an undo button?

I have started using a very simple and rough version of the Whatnext (my proof-of-concept nihilistic productivity planner) as my task list; it's marginally better than the Google spreadsheet I was using, but marginal is better enough when you want to eat a little dogfood.  The very second thing I did, after dancing with glee that I got something useful onto the internet in only 53 hours of work (plus 83 for getting the website back up and learning the basics of Django programming), was screw something up.  The third thing was to get annoyed because there's a confirmation step to deletion.

Why did I make it have a confirmation step?  I didn't—I followed the example for the tool I'm using, Django class-based views (I am continuing to blur any technical terminology in these programming essays so as to avoid scaring or boring).  I don't want to scrap the confirmation because, of course, I am terrified of deleting anything by accident.  Usability guru Jakob Neilsen argues that undo is more usable than confirmation.  More generally, I suspect that begging forgiveness is more usable than asking permission as long as forgiveness is easy enough to grant.  So, let's fix this by changing it from Confirmation to Undo.

Okay, easy enough, I'm plenty familiar with the "Soft delete" pattern, which is that you don't actually delete anything in the database, you just mark it as deleted and then exclude it from normal use.  And Django has a handful of packages to make it easy.  However, googling for best practices leads to this apparent ur-objection: The trouble with soft delete.
it usually ends up causing more harm than good
[Soft delete is s]imple, easy to understand, quick to implement and explain.
It is also, quite often, wrong.
Even more:
I just do not like this kind of design at all. I am firm believer of the architecture where only necessary data should be in single table and the useless data should be moved to an archived table.
Ugh.  I just want delete stuff without having to click twice and without risking losing something I wanted.  I want an undo, like Gmail:

(By the way, if you ever wished you could unsend an email right after you sent it, because it's only when you click send that your brain finishes figuring out what you want to say or not say and who you want to say it to or exclude, and you use Gmail, you should totally turn on undo.  I'm sure I use it more than once a week.)

I don't have time to read all of those links and reach an informed position in this debate.  I just want an undo button.  I keep browsing as a way of stalling, and see that someone else suggests yet another pattern: "the Command Pattern."  I start reading the link:
Command is a very powerful design pattern, whose intent is to encapsulate a request as an object, thereby letting you parameterize clients with different requests, queue or log requests, and support undoable operations.
Yeah, no, I'm not going to even read enough to understand what that means.  Okay, I want to finish dealing with the delete confirmation in ten minutes or less. And we are back to the classic programmer problem: should I do it the expedient way or the best way?  I think I'm going to compromise: I'm going to switch to Soft Delete for the moment, because it's expedient and a very reversible decision.  But I'm not going to build the undo yet, because that's going to take a while and I already have a task later on to figure out how to do the yellow notifications.  If I really need to undo I can duck under the covers.  And I've bookmarked all of those articles to read through and then consider which alternative to Soft delete, if any, is best for Whatnext.

I guess that's not a compromise; that's doing a quick and dirty partial fix to the immediate problem (which, if you remember, is that the delete confirmation annoys me) and punting on everything else.  In my next essay on this subject, I should probably tell you about the term "Technical Debt," because I just incurred some.  Meanwhile, go set up undo on your Gmail.

Saturday, November 15, 2014

In a better universe ...

In a better universe, putting two fitted sheets into a dryer together would cause their tangling powers to cancel out.

Against hidden passwords and masked fields on web forms

I hate this. This is bad design. I'm typing my Social Security Number into a site and it's masked. Because of the terrible epidemic of people sneaking into other peoples' homes and watching over their shoulders as they type their SSN into websites and then stealing their identities. Ok, sure, plenty of people are doing sensitive data entry in coffee shops and libraries and whatnot, but even there, I think the hassle of not being able to see what you are doing, especially when doing complicated data entry, outweighs the notional security of forcing all users to mask.  There is probably a Firefox extension to fix that.

One thing you can do in Firefox that is nice is, on poorly designed websites that prevent you from pasting in your password, reclaim the power to paste in your password.  See here.

My reward for filling out a long form (complete with yet another classic, silently truncating your entered password) was this:

HTTP Error 500
500 Internal Server Error
This server is incapable of servicing your request. Please try your request again later.
We apologize for any inconvenience this may cause.
If the problem persists, please contact a representative of ADP to assist you.

Friday, November 14, 2014

Things that have happened recently while programming

One.  I was typing in a terminal window and viewing the results of my work in a browser window.  I couldn't see any changes from what I was doing.  A common rookie mistake is to type into one environment and look for results in a different environment, such as working on the development server and checking the production server.  I'm no rookie so I have safeguards against this, like naming things "dev" and "production".  And just to be certain, I made a huge, deliberate error that would have broken things, and sure enough, I could see in the terminal window that things broke, so I could be completely sure that I wasn't typing on one server and viewing on a different server.  Do you see what just happened in that last sentence?  It took me about an hour to catch on.

Two.  I have a function that I have strongly suspected for a very long time should be recursive.  I just allowed it to be recursive.  It works better.  I feel relieved.

Three.  That function that's now recursive?  It just broke again.

Wednesday, November 12, 2014

Programmer problems: there's one right way to do that and it's wrong

I'm still stuck on that problem I discussed last week, how to get a few numbers from one place to another place within my problem.  Or rather, I've won one little learning battle but the programming war remained frozen like Stalingrad.  What I'm trying to do, task #190, is make my program so that when a user adds a book to the list of books they want to read (for example), the new book goes to the top of their book list but not to the top of, let's say, their recipe list.

In jargon terms, I'm trying to retain and pass around context, and apply that context consistently.  That context includes tags, of course: (once again, I'm blurring out all of the jargon) if I say "Books", the system should remember "Books" for as long as appropriate.  It also includes my identity: if I add a task, it should go on my list, not yours.  And the context should stick around: after adding the task, I'd like the website to remember to show me my list of tags, not yours or everybody's.

Last week I was stuck on one portion of this problem: taking the context from one web page, like after you click "Books" and "Filter", and passing it forward to the buttons on the page so that, when you add a new task, the controls to add a new task remember that you are concerned only with "Books".  Once I found the right tool (it turned out to be something called a QueryDict), I solved that in five minutes and two lines of code.  Of course, it had taken 10 cumulative hours to figure out those two lines of code, but ... they work, and they are the Right Way for my platform of choice.

But that's only covers passing the context around; I also need the system to properly apply the context when it stores and retrieves data.  And it stores and retrieves data for many reasons—when looking at the list, or when adding a new task, or when moving a task up or down, just to name a few—so I want to write the code for that once and have it work in all places.  This is called DRY, "Don't Repeat Yourself." Being the avid learner that I am, I went to find the most correct way to consolidate all of my repeated view filtering code in Django.  The instructions say to use a custom Manager:
You can use a custom Manager in a particular model by extending the base Manager class and instantiating your custom Manager in your model. There are two reasons you might want to customize a Manager: to add extra Manager methods, and/or to modify the initial QuerySet the Manager returns.
But when I go looking for examples, I stumble across an argument:
Using raw ORM query code in views and other high-level parts of your application is (usually) a bad idea. Instead, creating custom QuerySet APIs and attaching them to your models with a PassThroughManager from django-model-utils gives you the following benefits:
Although the very first comment is pure trolling.
There is no "right" way to use an ORM. Put it down and step away.
Troll aside, more googling finds that somebody agrees with the premise:
If you haven’t read that post, head over that way to hear why it’s a fantastic idea to build high level interfaces to your models, and because of limitations in Django’s manager API, to do so in QuerySet’s rather than Managers.
but has a different solution in mind:
While this definitely has the feel of slightly noodley trickery to it,  I think it starts to get at the core of what we might want in a more robust future ORM for Django: the ability to define high-level logic to assign meaning to particular collections of fields, and to preserve that logic across relations and chains.
Somebody else also agrees with the premise, but has a yet better way to do things:
But using Python mixins I've developed what I think to be a better way
(If there's one nice thing about online arguments about the best way to do a particular kind of computer programming, and there may not be, it's that everybody who wants to be read has to provide examples; nobody can just claim there isn't enough room in the margins to back up their claim.  On the other hand, you then have to read their examples.)

So the practical question is, do I want to do things the "rightest" way, which is going to mean researching this debate and picking a side, or do things the way the official documentation suggests, or do things the quick and dirty way and pay for it later? This is not a rhetorical question; being able to accurately estimate the cost of fixing it later is a professional skill which great programmers have. In my case, I'm trying to roughly balance expediency and rigor, and that's not working very well. I probably want to lean a bit back toward expediency.

Friday, November 7, 2014

Goodnight sweet Ivar

Go to sleep you little Ivar
Go to sleep you little Ivar
Come and lay your bones on the alabaster stones
And be my ever-loving Ivar

Thursday, November 6, 2014

When simple programming tasks become epic trials simply because you don't know what you're doing

I want to get some data from one part of my web program to another.  What I thought would be a trivial little task has turned into 7 and a half hours of work over the last week (counting blogging about it), which is not much shorter than the entire Lord of the Rings trilogy on Blu-Ray.  I'm going to tell you this saga because it is a very common happening with computer programming: you think you are (metaphorically) popping out for five minutes to drop your Netflix in the mailbox and grab some Mountain Dew and realize, days later, as you dodge orcs and hide from Nazgul, that you are guiding two hobbits with some Fingerhut ring and a stalker problem across Middle Earth and into Mordor.  This phenomenon plays hell with estimation, drives non-programmers batty, and goes a long way to explaining where there can be a 10x or 100x difference in productivity between two programmers.  We're going to go through my sad little example to try and give you, a non-programmer, a helpful peek into the abyss.  Don't worry, I'll blur all of the computer code.

Let's begin with a trailer featuring a scene from the third movie, when Sam and Frodo are dodging Shelob, the giant spider.  In my version, Frodo is played by the number 28, and Sam by 29.
<QueryDict: {u'notes': [u''], u'csrfmiddlewaretoken': [u'JJUhgrMzA3gtsMPnmzUsJYn4J6doONt4'], u'name': [u''], u'tags': [u'28', u'29']}>
See them stuck in Shelob's web?  I try to get them out of there using this magic incantation:
(Pdb) print self.request.GET["tags"]
And all I get is:
Obviously Sam can't go on without Frodo.  Our story cannot proceed.  Disaster!  Or, to abandon the increasingly tortured and confusing metaphor, I'm trying to complete Task 154, which visually is this:

This is a task from WhatNext, the nihilistic task management website that I've been progamming as a hobby, in between the knitting.  It's a task that has completely stalled progress for, as I've said, a week.

Our story begins with this innocent-seeming web page.  This is an example of the WhatNext task list:

See the little up and down arrows?  When you click, the task should move up or down.  And that works.  But.  I also have these nice tags, so that you can look at only a little bit of your list:

And I need to keep track of those tags for almost everything, or else things go wonky.  Which means, in the grossest terms, that that bit highlighted in the red box needs to go down where that red arrow is pointing.  AND IT WON'T GO THERE.

There are two kinds of programming: programming that does something never done before, and the other kind, re-inventing the wheel.  In a sense all programming is the first kind, because if something already does what you need, why not just use that?  There should never be a need for the second kind.  But in practice there are obstacles large and small.  For example, the program you need may be very expensive and you only need a fraction of what it does, so you take a cheaper program and try to extend it a little bit.  Or you have a Mac and the program that does what you need is only for Commodore 64.  This dilemma applies when you choose a program to use, and it applies when you are programming.  All programming is done on a foundation of what has come before; you don't typically rewrite an operating system, or email, or the alphabet.  And when you are creating a data-backed website, you are writing the same kind of program that everybody's been writing for two decades, which is only a slight change to client-server programs for decades before that, all the way back to mainframes and terminals.  So most of the idioms and patterns and best practices have been codified long since, in the form of standards but particularly in the form of languages and platforms and libraries.  One need only choose and master a platform in order to start out, in theory, 90% finished, with only the particulars of your situation remaining to be programmed.  So programming is typically not inventing a new solution to a problem; it's figuring out the most efficient ways to apply existing solutions to old problems for new customers.

Which is why it's so unbelievably frustrating to get stuck trying to do something in Django so obvious and common as taking some parameters from the incoming URL and sticking them in the outgoing URLs so that the user does not lose their metaphorical place when they click on things.  There are some best practices to handling URL parameters, to prevent problems like mangling the spaces between words, or allowing a Ukranian rebel hacker to steal your credit card and denouce the West on your home page, but those just make it more likely that Django (the platform I've chosen for my program) would have this functionality built it. Which means that I've spent most of the 7+ hours googling for examples that are close to my problem, solve a piece of my problem, but don't quite solve all of it.  Django provides many, many ways to do anything.  Which means it doesn't provide a right way to do something, only ways with differing degrees of appropriateness.

Let's get specific.  First, I have to capture any data the user provides in the URL.  That is, if the incoming URL says that tags 28 and 29 are in use, I need to store "28" and "29" somewhere.  Then, I have to make sure they are really plain numbers, and not some cryptic bit of code that tricks my computer into divulging the launch codes.  Then I have to prepare them for the outgoing URLs, which in this case means turning them into the sequence of characters &tags=28&tags=29.  And then I have to stick that sequence of characters at the end of all of the URLs on the page, such as the "Move Down" button and the "Move Up" button.

I know how to do almost all of this. I've done it before in this program, everything but sanitizing the input, and I've glimpsed bits of that so I'm confident I'll find tools for it in the platform.  Nothing here should take more than five minutes, so I'll round up and say the whole task ought to be done in an hour.

First, I have to catch the data. The standard place to capture incoming variables in Django is  However, this makes it clear that I can't use it because the technique is not appropriate for parameters that "chang[e] the way the resource is displayed".  Another programmer might decide to go ahead and use this technique anyway, and that might work for them, or they might hit a different snag later.  Or it might work now but then cause a problem for the next programmer in a few years, when something else changes.  I'm going to play it straight and try to find the preferred way to do this.

So I look for another way to catch my 28 and 29, and find it, but when I put that code into my program, nothing happens.  Nothing happens is probably the second-most common outcome of any change to a computer program, beaten only by error messages.  Why doesn't anything happen?  I Google, and find another question along the same topic, and the answer should clarify some things for you:
... they are at the class level and are class variables. As for the NameError, where are you trying to do year = self.kwargs['year']? You should be doing it in a method, you can't do it at the class level. 

I think that that means that you can't use these particular commands in the main body of a "class", which is a particular type of mini-program within my program (which, let's recall, is itself inside of a another program, called Django, which is effectively inside of another program, called Python, which ... reminds me of a movie called eXistenZ.

Do you remember that movie? SPOILERS! It came out the same year as the Matrix and The Thirteenth Floor. In the Matrix, it turns out that SPOILERS our reality is just a computer simulation. In The Thirteenth Floor, researchers get stuck in their computer simulation, but when they get back to reality it turns out that they in turn are just part of SPOILERS a computer simulation. So that's three layers to the Matrix's two. Add a layer of complexity, shrink the box office by an order of magnitude. In eXistenZ, you simply lose count of the layers of reality, so it didn't make much money at all. In Python, each bit of program tends to run inside of another bit of program, and you can interrupt the program and run a command to show you all of the layers of program that are in effect, so that you can track down problems. Here's what you get with my program at the point of error:
(Pdb) where
-> self.__bootstrap_inner()
-> self.__target(*self.__args, **self.__kwargs)e
-> self.finish_request(request, client_address)e
-> self.RequestHandlerClass(request, client_address, self)e
-> super(WSGIRequestHandler, self).__init__(*args, **kwargs)e
-> self.handle()e
-> self.result = application(self.environ, self.start_response)e
-> return self.application(environ, start_response)e
-> response = self.get_response(request)e
-> response = wrapped_callback(request, *callback_args, **callback_kwargs)e
-> return self.dispatch(request, *args, **kwargs)e
-> request, *args, **kwargs)e
-> return handler(request, *args, **kwargs)e
-> context = self.get_context_data()
-> /home/ao3/ao3/whatnext/
return context

Each -> is another layer of context, a bit of previously existing program that is part of the platform of Django and does some essential bit of the process of serving up a database-backed web page. So part of my time is spent checking the most superficial layer, and maybe one or two layers below that, to see what's going on and look around for where 28 and 29 might be. But I also spent a few hours (not included in the 7.5) getting the development site working on my laptop, so I could work on this in coffee shops and airports, and then on my laptop this particular debugging tool wasn't working right, so that was another thirty minutes fussing. And as part of getting the site running on my laptop, I went to use a backup copy of the data to put on my laptop and realized that I didn't have automatic backups running, so that was another thirty minutes to get that going, and if I'm conscientious, another 30 minutes in a week or so to make sure backups are actually happening.)

So, three hours of parathentical work later, I figured out from reading the internet that the command that retrieves 28 and 29 has to be used inside of a method, which is a mini-program inside of the mini-program. So, that's a hurdle well cleared. Now, I have to figure out which mini-program in my mini-program is the right kind.  That is to say, of the many mini-programs that Django developers have added and made idiomatic, I have to figure out which one they intended to be used in this circumstance.  In this case, I *think* it's get_context_data, because of yet another helpful question.

Also, at this point one is generally tense, bored, and frustrated, and a little stupid, so something like "Don't mix *args and **kwargs in call to reverse()!" becomes very funny: I guess although *args and **kwargs get along fine in normal conditions, in reverse() they fight and, since the **kwargs are bigger and have more asterisks, they tend to eat all of the *args.  (That's a kwarg to the right, although it may actually be Kwarg itself.)

So I think I know which mini-program I should put my commands in to get 28 and 29, but then I read that Django’s generic class based views now automatically include a view variable in the context, so maybe I don't need that first step at all, that's already done automatically and I can skip to the second step, which if you'll recall was to make sure they are safe, innocent numbers? Except that, before I can move on to that, I get stuck because when I ask for my data in exactly the right place, using what I think is the right magic command, I just get one number, not both. You may remember this scene from the trailer.

(And, to be completely clear, 28 and 29 are just stand-ins for many possible numbers, depending on what a user enters.  I'm just assuming that, if I can get 28 and 29 through to where they need to be, it will work for other numbers.  But I'm lucky that I picked two numbers, not one, to be my sample data, because if I'd picked just 28, for example, then I never would have noticed that a hypothetical second, third, or later number would just disappear.  That is, I wouldn't notice until weeks later, when I had already been using my program and any task that had one tag would move just fine, but any task with two or more tags would, when moved, go to the wrong place, and how many times would that happen before I noticed, much less diagnosed the problem?  What other potential problems am I not uncovering now because of my choice of 28 and 29 as my working test numbers?)

The reality check that keeps creeping into my brain is, surely many people have solved this exact problem before me, using this exact program?  I'm not doing anything special.  Why can't I find any giant's shoulder to stand on?  So this becomes not a programming problem, but a problem of finding the right query to lead me to a previous example.  Finally, I try "django get list from querydict", which leads to a clue (emphasis added):
QueryDict.iteritems() uses the same last-value logic as QueryDict.__getitem__(), which means that if the key has more than one value, __getitem__()returns the last value.

Which is helpful, because it's talking about exactly the place where I'm stuck. And, more importantly, it's clued me in to the fact that the tool I'm looking for, the thing built in to Django to make this problem easy, has been right under my nose the whole time. It's QueryDict. I haven't solved my problem, but I've found the right tool to use, so I guess I'll just go and read up on that now. And this, amigas and amigos, is what may be happening when your programmer goes off to do something that even they think is simple but, days later, when you ask how it's going, they just snarl and throw cheetos at you. 

This story also illuminates the discrepency in performance between different programmers.  I know in fairly specific terms what I want to happen, but have no idea how to make it happen in this particular platform, so I'm spending all of my time learning the platform, painfully.  A "1-hour" task is shaping up to be 10 hours or more.  Another programmer might not know this platform but might have developed much stronger instincts for data-backed web idioms, and would find the tools they needed much more quickly.  When I complete Task 154 and commit the new code changes, I expect there to be maybe 10 or 20 new lines of code.  Someone completely familiar with Django might have solved this task in literally the time it takes to type out those lines, no extra thought or research required.  Someone fairly familiar with Django but sloppy might have solved the task in 5 minutes but also have added in the bug we spotted, where moving a task with two or more tags puts in (silently) in the wrong place, because 28 and 29 turns into just 29 but they don't notice.  All I know at this point of Task 154 is that I'm too ignorant even to estimate how much more time it's going to take me.

Good rappin' at you, amigos and amigas.  If you take nothing else from this blog post, I hope you take with you a wonder at why you read this far.

Monday, November 3, 2014

San Franciscans - go see the Touré-Raichel Collective this Saturday night

I saw them last time they were in San Francisco and it was the most pleasing live music I've experienced in a very long time.  Buy tickets right now and then go listen to a little bit to whet your ears.  My favorite part from the live show was when Raichel (piano) and Touré (guitar; he's the son of Ali Farka Touré) had a musical conversation: you can sort of imagine Dueling Banjos but, instead of challenging and repeating, it sounded to my barely musically educated ears like they were having a conversation, or even telling jokes, and it sounded lovely even when I didn't get the jokes, but a few of the jokes I understood, or imagined I understood, for example an arpeggio to be a punchline, and I laughed a little bit too.  And then the calabash player and the bass player did the same thing and were just as good and funny.

(Also: Seattle on Nov 15 and Toronto on Nov 21)

Sunday, November 2, 2014

Hat 38 is done

Thanks to the bizarrely warm summer here in San Francisco, not too many heavy wools hats have been warranted.  But this one is done and today was a bit chilly.

The Ravelry project.

Saturday, October 25, 2014

Something you can do today to make the US better

I've been contributing to Lawrence Lessig's quixotic attempt to get corrupt money out of US national politics by raising money to support reformers and remove the worst politicians.  Mayday.US's effort in Michigan's sixth Congressional district is apparently not totally quixotic, because the corrupt incumbent is reportedly threatening some of Lessig's big-money donors who have business pending before his committee.  That seems pretty straightforwardly corrupt.  Read Lessig's post.  Then give some anti-corruption money.

Friday, October 24, 2014

Air travel means more hats

Finally sewed up the equilateral triangle hat:

And made a new hat.  I brought a few new stitches to try out, and ended up with this twist stitch diamond pattern.  It seems like a big pain at first because it's all togs and knitting through the back loop, but once you get the hand of the left and right twist stitches they get easier and faster.  I'm finishing it off with a little tail; not sure how long it'll go.  I thought it would be too big for an infant but it's quite dense and not eager to stretch

Ravelry pattern for twist stitch hatRavelry project for the equilateral hat.

Saturday, October 18, 2014

Less popular, but still equally valid, signs of the coming apocalypse.

The increased use of the word "training" as a countable noun, as measured through the published use of the shouldn't-be-a-word "trainings".

Tuesday, September 30, 2014

Programming is understanding the user's problem and deciding how thoroughly to solve it.

I got stalled working on my program.  So I want to write about the nuts and bolts of programming and how I got stuck and maybe that will unstick things. I'm writing as a hobbyist programmer but as a professional project manager and therefore as a professional goer between users and programmers, and if you are a user struggling to understand how the sausage is made, or, more likely, why it takes so long, this may shed some light.

Programming is writing a program that spans the gap between what a person wants a computer to do and the underlying mechanical functions of the computer.  Our electronic computers can really only do one thing: receive two signals, and, if either or both are one, output one, otherwise output zero.  But in decades since electronic computers were invented, programmers have built up a vast library of programs and conventions that abstract away the ones and zeros, so that the building blocks of my program are concepts like form and field length and web page and password.

So when one programs in 2014, the specific program to write is driven by two questions: "what is the user's problem?" and "how thoroughly should I solve their problem?"  Let's analyze my stall with these questions in mind.

I'm working on Task 155, Integrate "Add Task" into table row.  In Scrum terms, implementing this story: "As someone looking at my list of things to do, I want to add a new thing directly to the list, so that I don't forget my new thing or lose my train of thought."  I've already put 11 hours of programming into this project, so I have some stuff I can build on top of.

You can see six tasks in the list, and then I've already programmed another row with a few boxes, in which I've already typed the name of the next task, "buy some milk", and the estimate, "ten minutes".

So the user need is pretty clear: in abstract terms, add a new task quickly.  In concrete website vocabulary, add a new task by typing just the name and maybe the estimate, clicking a button, and then seeing the list updated with the new task in place.

How thoroughly am I solving this?  Well, I've already completed Task 129, Basic Tasks, which included features to create, read, update, and delete tasks.  So I'm hoping to re-use that work to make Task 155 very easy.  A thorough solution with very little new work.  How is this possible?

Entering data into a computer has a number of common complications: security, validation, error handling, and the like.  I've built the whole web program on top of another program called Django, which provides all of these features automatically.  I already defined all of my data for Task 129, so Django knows which fields are required, which fields have special rules ("Wait until" must be a date), and so forth.  This is called the data model.

The "Create" portion of Task 129 is basically "As a person with a task, I want to record everything I know about this task so that I can track and manage the task later."  In concrete web terms, this means a form.  So that's one web page this displays all of fields to be entered, some code to handle receiving and processing the input and a page to display any problems and let the user try to fix them, and something to show after the input is accepted.

With Django, most of that can be done thusly:
<form action="" method="post">
    {% csrf_token %}
    {{ form.as_p }}
    <input type="submit" value="Done">
which produces this form (shown partially filled out):

This code handles, or rather invokes much more Django code which handles, most of the complexity of full-featured web data entry.  There's a hidden token to prevent certain kinds of security breaches; each different type of data is being collected in the appropriate kind of web input field, from text to choices to paragraph boxes.  And validation and error handling

There are a few other bits of code here and there, to do things such as determine the URL of this page, make sure the user is logged in, and so forth.  But using the feature Django calls Class-Based ModelForms, I can write what would otherwise be hundreds of lines of detailed code in maybe a dozen or two lines.  So, I'll just re-use that for Task 155.

Or so I thought.  The problem with using a program someone else has created is that it may not solve your exact problem.  The best programs are flexible and handle many variations on the problem.  This code is really code at displaying a form on one page, and handling form submission.  But if you go back to the first screenshot, you can see that there is already a form on that page.  And, I don't want the whole form, I only want two fields; and I don't want them line after line, I want them fitted into one row on the table, matching the column headings.

Okay, so I can't use the command {{ form.as_p }} to display the form, so I'll have to write out the form display code myself.  Then I can have the form go to the form submission code, and if it works, come right back to this page with the new task inserted.  It will require a full page refresh, rather than magic web 2.0 stuff where the input just appears, but it should be a good first step.

More complications: {{ form.as_p }} is dynamic.  That is, every time somebody browses to a web page with that code, those four lines of code above turn into this HTML code (which you should scroll past without reading):
<form action="" method="post">
  <input type='hidden' name='csrfmiddlewaretoken' value='4U1ynhzn9GTRDxoynhYnw2JWfXsCqB6P' />
    <label for="id_name">Name:</label> <input id="id_name" maxlength="500" name="name" type="text" />
    <label for="id_task_order_0">Order:</label>
    <ul id="id_task_order">
        <label for="id_task_order_0">
      <input id="id_task_order_0" name="task_order" type="radio" value="-1" />
    <label for="id_task_order_1">
      <input checked="checked" id="id_task_order_1" name="task_order" type="radio" value="11" />
    <label for="id_estimate">
    <input id="id_estimate" maxlength="50" name="estimate" type="text" />
    <label for="id_wait_until">
      Wait until:
    <input id="id_wait_until" name="wait_until" type="text" />
    <label for="id_notes">
    <textarea cols="40" id="id_notes" name="notes" rows="10">
    <label for="id_tags">
    <select multiple="multiple" id="id_tags" name="tags">
      <option value="2">
      <option value="3">
      <option value="5">
    <span class="helptext">
      Hold down "Control", or "Command" on a Mac, to select more than one.
  <input type="submit" value="Done">

I can manually write some of that to fit into the bottom row of my table, because the person who wrote the fancy table display program that I use to make my table look not totally ugly made a provision to put stuff into the bottom row of the table.  But I can't just put in the HTML code that I need, because there's a security measure in Django forms to prevent "csrf" attacks, and so Django needs to create a custom code for each form.  So I can't use the Django code that magically solves my problem because it doesn't solve my exact problem (the people who wrote the form program didn't provide for creating the HTML code as a row in a table), and I can't use the next more primitive level of abstraction, the HTML code, because I still need some functions from the Django code.

The quick and easy win-win solution to the user's problem just got slow and dirty, but I have to get something working so I hack out the HTML for the more limited form that I'll squeeze into the table row:
<tfoot>   <form action="create/" method="POST">     {% csrf_token %}     <tr>       <td colspan="2">         <div class="fieldWrapper">           <input id="id_name" maxlength="500" name="name" type="text" />         </div>       </td>       <td>         {{ add_task_form.task_order }}         </td>       <td><div class="fieldWrapper">           <input id="id_estimate" maxlength="50" name="estimate" type="text" />         </div>         <td colspan="6">           <input type="submit" value="Add new task"/>         </td>     </tr>     </form> </tfoot>
But, if I ever reword "estimate" to, let's say, "Estimated Time to Complete", I can make that change in one place, the data model, and it will automatically take effect everywhere that the Estimate field is used except for this hacked code.  By solving the user's problem more expediently, I've damaged the thoroughness of my solution.  And this code doesn't include the dynamic csrf token; I can either disable that security feature (harming the thoroughness) or go back to the drawing board (delaying the solution).

There's another problem.  When I planned out the Create function of Task 129, I had the problem of what to about the new task order.  This is a pretty typical sequence of events in programming: the user specifies a need, the programmer understands it and starts to program a solution, and runs into a complication, most often forced into view by the already-completed data model, and realizes that the need isn't specified thoroughly enough.  What that looks like here is, the data model requires that every task have a unique, non-null task order.  That is, every single task must always have a place in the stack ranking and no two tasks can ever by tied.  So what's the task order for a brand new task?  The user just asked that it be possible to create a new task.

At this point, a programmer either goes back to the user, who may not understand what's missing: "What's a task order?  I didn't order anything."  Or they may not be able to articulate what they want: "Make it so that the next task is next.  Not at the top or bottom, just next."  Or may not be able to understand why what they want is impossible: "Just leave it blank for now.  No, I don't want it to be tied with anything; just leave the order blank so it can be sorted later.  No, I don't know what relaxing the field integrity means but it sounds bad so don't do that.  Just figure out a way that it can be blank but still all tasks are in order, no ties."  Or the programmer just guesses what the user would want, or what might make the user happy.

In my case, since I'm the programmer and the user, I know what I want to try for now: all new tasks must go either first or last.  I have a suspicion that most people who maintain task lists are either all one or all the other: either you put your new tasks at the top and you are always fighting fires and never finishing things, or you put them at the bottom and basically it takes forever and a nag for anyone to get anything from you because you are always finishing something someone asked for a long time ago.  Of course it makes sense to be able to put tasks somewhere in the middle, and to be able to move them around, but I'll deal with those cases later.  And, if I've finished tasks ordered 1, 2, and 3, and am working on a task ordered 4, and add a new task at the top, should that task go before 1 or between 3 and 4?  All to be dealt with later; in order to finish Task 129 and be able to create tasks at all, I decided to allow only absolute top or absolute bottom.

And I had to write some code to do that.  What I did was, in the place in the program where Django prepares the form for display (i.e., renders the code form.as_p into dozens of lines of HTML as shown above), I found how the programmers who created Django intend for Django programmers to customize the process.  And I made it so that, at the moment the user clicks "New Task" and Django figures out the web page to show them, Django figures out what the high and low orders are, so that those two buttons are tied to the numbers.  That is, if you have four tasks, ordered 1 through 4, Django will make "top" equal to 0 and "bottom" equal to 5.  Of course, to do even that I had to figure out how to use Django's database code to get this information, something I already know how to do directly in SQL but I can't do that or I'll break Django's database-independence, which means that if I ever publish this code, other people won't be able to use it on the database of their choice.  As with the hacked HTML, I can do it quickly, or I can do it more thoroughly.  I spent the time to do it correctly, and learned some Django query syntax:

lowest_order = Task.objects.filter(user=user).order_by('-task_order')[0].task_order + 1 highest_order = Task.objects.filter(user=user).order_by('task_order')[0].task_order - 1
But when I display the hand-coded HTML form in the row of the task table, I skip these calculations.  In fact, I don't include the task order field at all.  So when the user clicks the Add New Task button in the row and the data is passed to the Django function that processes form input according to the data model, it rejects the input because I specified that task order can never be empty.  And instead of seeing the task list with the new task included at the bottom, the user sees the full form page for new tasks, with an error message that doesn't make much sense:

It's also hard to spot since I haven't applied any formatting to the raw HTML output, but that's a different Story.  And it says "this field" when "Order" would be clearer, but that's yet another Story.

So, in case you forgot what problem we were trying to solve and whether or not we've solved it, a recap: I am trying to allow the user to input a new task right in the task table.  To do this I tried to reuse functionality I had already programmed for adding new tasks via a form on its own page, but when I tried to squeeze that form into the bottom row of the table, the functionality I was re-using wasn't suitable and I had to hand-code some stuff.  And, I discovered that the user need itself was inconsistent: the user wants to create a new task without specifying its order, but elsewhere the user has insisted that all tasks be sequentially ordered, and the user hasn't specified how to reconcile this conflict.

So.  I could make a rule that all tasks added in this fashion go to the bottom of the task order.  After all, the row for adding tasks is at the bottom of the table.  But to do so and still reuse the form submission and processing code, which requires a task order to be provided, I would have to dynamically figure out what the bottom task order should be, and figure out how to insert that dynamic information into my static HTML.  And, this isn't even a good idea, because you have noticed that this approach introduces a kind of weakness called a race condition

If the user clicks "New Task", the system calculates top and bottom task orders and bakes those into the New Task form and displays that on a page.  Suppose the user does this on their computer and then gets distracted by a phone call and wanders away.  At the end of the phone call, they have another new task, so naturally they try to enter it on their phone.  When they browse to the site on the phone and click New Task, the system again generates top and bottom task orders.  They finish creating the task, then go back to their computer, remember the original task they were trying to create, type it in, and submit.  The original web page on the computer and the new web page on the phone each generated the same top and bottom numbers, so if they picked, let's say, "top" for both, when they click to save on the computer it will try to save the task to an order that's just been used, and it will fail with an error.  So that's not a very thorough solution.

So.  My attempt at quick and easy re-use is stymied, and I've discovered a flaw in my existing code: highest and lowest should be calculated at the last possible instant, not the first.  And, it occurs to me that using integers to store task order is a bad idea because each new task added in the middle will require changing the order number of half of the existing tasks.  I thought about using floating point numbers (so that new task could be 1.5 or 1.25 etc) but that might open up other worm cans; I should maybe research best practices on this.  In trying to solve a new user problem, I revealed weaknesses in an already "solved" user problem without making much forward progress.

There's lots more to say about the interplay between understanding user problems, trying to solve them, and deciding how thoroughly to solve them.  My solution to Task 129, the basic elements of adding a new task, turned out to be so incomplete that I couldn't use them for some things I was already pretty sure I'd be wanting to do.  But on the other hand, I'll still be able to salvage most of that solution, and re-use it for other problems, and it was a morale boost to get something working.  More to write on this in the future.  If you work with software developers, I hope this gave you a little insight into why everything seems like such a big deal.  Blogging this out helped me unstall myself:

I need to go back and re-solve 129 (by switching from early to late binding of task order, and maybe simplifying my solution so that all new tasks are always last, taking away the user choice.  I need to research changing the data model from integer to floating point task orders, by finding 2-3 examples of other systems and seeing what they do.  And then I need to bite the bullet for 155 and figure out how to build the initial form dynamically, which means figuring out how to get Django to do two forms on one page.  I should mention that I did try that already, and ran into the biggest productivity killer of all: I did what seemed like should work and it just didn't, and now I have to figure out why.  Which is the actual biggest element of programming: debugging.