Tuesday, March 17, 2015

Programming as a series of tiny a-has

One fundamental issue in project planning is big vs small.  Estimating and tracking tasks generally requires them to be small.  Once its component tasks are broken out small enough to work with day to day, any non-trivial project will have hundreds or thousands of tasks or more.  It's hard to work with a list of a thousand items.  It's hard just to read a list of a thousand items.  That's why Top Ten lists are much more popular than Top One Thousand lists.

In WhatNext, my experiment in nihilistic project management software, I am playing with a few different ways to manage tasks.  I want to be able to navigate from literally thousands of tasks, everything from simple one-time TODOs to books to read to more complex things like episodic TV shows to watch or recurring errands to remember, and get down to a list of about ten relevant and appropriate tasks that I might choose to do right now.  How?

Filtering is one obvious method: I want to relax, so show me only books and movies.  Or filtering by modality: I'm on a desert island, so only show me things I can do without internet.  Prioritizing is another: show me only the top few items.  Grouping is another, and the subject of this post.

There are many ways to group tasks; one essential decision is, can groups contain other groups?  Some tools allow a few fixed levels of depth, such as
  - Task
    - Sub-Task
while other tools allow indefinite hierarchies.  Either way, tasks are generally grouped via hierarchies.
I. Cook Dinner for Children
  1. choose a recipe
  2. obtain ingredients
     a. Make list of what is not in pantry
     b. go to the forest
     c. gather items on list
  3. cook the recipe
  4. set the table
  5. let the children out of the dungeon
II. Conquer the world
  1. watch all episodes of Pinky & The Brain
  2. pick the best strategy
  3. implement the best strategy
I have a few qualms about task hierarchies as they apply to task lists.  The complex structure of a hierarchy complicates using filtering and sorting to narrow down a list of thousands to a list of ten.  And in terms of estimation, the list is a bit redundant: Item I contains exactly the same work as items I.1 through I.5 (and I.2 is the same as I.2.a + I.2.b + I.2.c).  Estimation can be top-down or bottom-up; if I have estimated both ways then adding everything up leads to double-counting.  Should I use the top-down estimate for I or the bottom-up sum of the rest or somehow merge the two kinds of estimates?  From a data modeling perspective, it seems unclean; even if I can't say for sure how I'll use this data, I don't want to build on the foundation of a flawed model.  In fact, that's even more important when I can't say for sure how I'll use the data: if the model is more accurate, it'll probably hold up better in unforeseen circumstances.  So let's do some modeling.

A task is a piece of work to be done, but think about it in context of WhatNext's motto, "Be very, very humble, for the hope of mortals is worms."  Why are you doing this task?  So that after the task is done, the state of the world is different.  After the task is done, you will be living in a world in which your children are well-fed (for now), your laundry is clean (for now), your boss is mollified (for now).  So a task is implicitly two things: a piece of work, and a desired outcome.  This gives us a method for task breakdown: keep the parent task's intended outcome, and replace its piece of work with many smaller pieces.

So "Cook Dinner for Children" becomes
outcome:           the children are nourished
constituent work:  choose a recipe
                   obtain ingredients
                   cook the recipe
                   set the table
                   let the children out of the dungeon
This may seem like an overly elaborate way to break out a task; what problem does this solve?  One problem this solves is how to have a single prioritized list of heterogeneous tasks: tasks of wildly different sizes, with sub-tasks of different goals inter-mingled.  Consider this simple task list:

With this task list, all of task 1 has to be done before task 2 can be started.  What if you need to make progress on both tasks?  Convert these tasks into outcomes, break out work into smaller tasks, and you have a chance to mix things up a little bit, sneak in a little progress on the second desired outcome before necessarily finishing all the tasks for the first outcome.

Something that often happens in the real world, however, is that even before you can start on your top task, new things get added to your list.  WhatNext assumes that new items go either at the top or bottom, usually at the top, so you could quickly end up with this list:

Here's another version of that table, trying to simplify by only showing the outcome as a detail of the final relevant task.  (I'm still struggling to find a pretty way to show both Outcomes and Tasks in one list.)

Still ugly, but let's go with it for now.  Anyway, we are finally ready for the new feature I've been working on.  Suppose I decide that "Nourish children" is the most important and urgent outcome on the list, and I want to get it done as soon as possible, even before things that I briefly thought I had time to do first.  How do I do that?  The simplest way I can think of is to add a button to each Outcome to move it to the top.

And now we get to the essence of "business analysis" or "requirements gathering", or step 1 of programming.  What exactly should happen when the arrow button is clicked?  Let's say, all of the tasks for "Nourish children" should move above all other tasks in the list, while preserving their order relative to each other.  I think.  (What if there are snoozed tasks that we aren't seeing?  When they unsnooze, what will their orders be?  What about closed tasks, are they affected by this re-ordering?  etc etc.  All fun details to consider.)

And now the specific tiny a-ha moment that I wanted to share with you. How exactly should the tasks be moved to the top?  In WhatNext, task order is internally tracked using a number between 0 and 1.  So what's really happening behind the scenes is:

The second step in programming this is to translate the human-language expression into a slightly more algorithmic expression, in some kind of pseudo-code.
1. make a list of all tasks associated with the selected outcome
2. move each item on the list from step 1 to the top of the master list
But if you go and implement that naively, you may get something like this:

See the problem?  When the Nourish children tasks were moved up one at a time, top one first, each succeeding one went on top of that, so ultimately their order was reversed.  So we need to refine the algorithm a bit:
1. make a list of all tasks associated with the selected outcome, ordered by internal sequence
2. for each item on the list, considered in order:
  1. identify the first task in the master list 
  2. change the internal sequence of the item to a number
     1. just smaller than the interal sequence of the first task,
     2. but also, if this is not the first item on the list,
        to a number just larger than any previous 
        just-resequenced tasks
And now we translate this into  the relevant programming language.  By which I don't mean just Python or whatever programming language one is using.  The "language" available includes everything available in the platform, any added libraries, and any functions one has already written.  Happily, I already wrote a function that can do all of item 2.1,  get_task_order. But we still have to handle all that stuff in 2.2.2 about remembering which task we are moving so that we don't scramble the order of the tasks.  And here is the tiny a-ha: If we reverse the order in step 1, we can skip all of step 2.2.2, because the whole problem with step 2.2.1 is that it reverses order, so just pre-reverse it in step 1 and 2.2.1 will reverse it a second time. (Caveats: make sure that nothing else changes in the middle of this operation. And a test case that only had one task wouldn't have revealed this problem.)
Since I don't need to add any new functions, the Python/Django code is quite simple:
outline = Outline.objects.get(id=pk)

tasks = outline.tasks.order_by('-task_order')

for task in tasks:
    task.task_order = get_task_order()
These tiny a-has are, I suspect, one of the sources of the addictive pleasure of programming.

No comments :

Post a Comment