My History of Task Management Systems

Both at home and in the office, I have a huge pile of tasks that would like to be done. Their priority varies widely from an idea to create that neat program, not knowing whether it will be useful at all, up to a broken file server in a branch office, blocking the work of 30 users. This pile of tasks is more than I can reliably keep in my sieve-like head, so I started taking notes.
My goal was (and still is): When I am ready to start a new task, e.g. in the morning or when I've finished the previous task, I want to look into the system and see clearly what I should do next.

My History of ToDo Management Systems
By mid 1993, after 2 years in my current job the office, the task pile was about 5 cm of 9x9 cm sheets, one sheet per task, each marked with importance (A,B,C), urgency (1,2,3) and the date I was asked to do the task. Most of the tasks were in B2, so the prioritisation was not much help. I got the recommendation to work "first come, first served", but due to the size of the pile, the waiting times were inacceptable. So when new tasks came in (or I got external reminders of old ones) that could not wait 4 months, I started to assign them a higher priority. Because very few incoming tasks could wait for 4 months, many of them got a higher priority - until the higher priority queue became inaccepably long too.
Around 1995 I experimented with a self made database application trying to combine task queue management and timekeeping. I really had to program a function "bulk priority reduction" to clear the highest priority queue for the really urgent tasks.
Moving to a Psion pocket computer did not help much, because it only turned the pile into a file. 5 priorities and a date weren't enough structure. Every few months, usually during a long train ride, I had to look through a few hundred tasks, deciding the fate of each of them. Priority 1 could expect work really soon, priority 2 was usually at the head of the queue being worked at, priority 3 and lower never had a chance, except by being bumped during a review session.
In the summer of 2003, after having worn out several Psions (Series 3, Series 3a, Revo, Series 5), I bought a Palm Tungsten C. The built in task manager was even smaller than Psion's, but there was a freeware program called Progect. It was meant for project management and could arrange tasks (or any other notes) as a tree. I used Progect for everything, not just task management. I've never seen a pocket mind map system coming close to the efficiency of Progect. It enabled me to arrange the tasks in a tree, so I could quickly look through related projects, e.g. for a certain branch. For main office work ("what's up next, regardless of the branch"), there was a "flat view", showing all tasks meeting certain criteria (e.g. date not in the future, priority 3 or better) in a selectable order (e.g. sort by priority, then date, ascending). But because even Progect only had 5 priorites and a date, there still was the "barely structured pile" effect.
By late 2007 I was frustrated enough, that I started multiple intensive searches for PC based task managers. I found websites describing the "Getting Things Done" style, which appealed to me, especially the part "get stuff out of your head" and the definition of physical actions. During the last days of December 2007, in the free time "between the years", I discovered ThinkingRock, then at version 2.0 epsilon. On the first glance it had an overwhelming amount of features, but after half a day of reading and trying, I started to like them:

• Projects, subprojects and tasks can be deeply nested. This enabled me to keep the tree structure of Progect, which closely fits my way of thinking. I can freely move tasks and subprojects within the parent project; the sequence is remembered (this is in contrast to other task management programs I've tried).
  
• I can define my own set of priorities and give them names instead of just numbers. This helps in assigning priorities more consistently.
• There are a number of properties per tasks that can be used for sorting and filtering: context, topic, energy, time (duration), date created, start date, due date, state (inactive, do ASAP, ...). Each topic can get its own background and foreground colour.
  
• I can set a project as "sequenced" and create a number of tasks. I need to set only the first one as "do ASAP" and can leave the others as "inactive". When I mark the last "do ASAP" task as done, the first inactive task automatically becomes "do ASAP". This is not a full dependency mechanism, but good enough for many cases.
  
• I can define and refine action screens, which are flat lists of the tasks in the tree. My favorite view (only a small adapation of the standard view) was: filter to tasks not yet done, status "do ASAP", without or past start date, order by priority. The tasks of same priority were ordered by the sequence they had in the project tree. So if I arranged the projects by importance, I had a good sequence to work at. Finally the "3 priorities and a date is too little structure" problem had been solved.

I 'm writing this last sentence in past tense, because an update (I think it was 2.0 epsilon to 2.0.1) broke that important feature "last sort order is by project tree sequence". That almost broke my life, because I was back again to "5 priorites and a date".
During a beautiful holiday in the mountains of Berchtesgaden (the southeastern corner of Germany, near Salzburg/Austria) I recalled my boss speaking of different priorities of different branches. So I decided to introduce a separate topic for each branch and to use the topic as a second sort criterion. With "date created" as a third criterion I was back in business again and I'm happy for almost a year now.


My Dream
I still thought, that there must be a way to assign priorities in a way that the head of queue draws a meaningful line between important and non-important stuff. So I need much more than 3 priorities - and a way to assign them consistently.

Trying to decide on a technology - Phases of Documentation Use

So what should I use for documentation purposes?
Probably the answer is "it depends". But I want my ideas scattered among as few places as possible, so I don't want to use every possible technique for every possible thing. Because there won't be the single ideal solution, I will have to live with multiples techniques and thus multiple places of documentation.

My idea is, that the requirements of a documentation system vary by the project phase it is used in. I think in these phases:

1. Idea

Somthing comes into my mind. I want to make a very very quick note, faster than the fading time of my ultra short time memory, which is somewhat between 9 and 15 seconds. Booting a netbook would take too long, so I grab my pocket computer, start the note program and jot down a note of 2 or 3 lines (on my old palm this was even faster: push one single "note" hardware button, hit "new note" and I'm redy to type). This happens to me dozens of times per day.

When I have enough concentration and the reach of a "real" computer, I want to put these notes into a system, so they are arranged and findable.
Usually there is no sharing yet, but a large need of portability. Often enough, those concentrated times are away from home or office or network connectivity. So on-line techniques are unsuable in the idea phase, which means that blogs and wikis are out of the game. HTML files are too hard to create, too hard to rearrange and too permanent - then tend to hang aroung in the intranet for years, which is inadequate for an idea I have dropped a few months after inception. A good creation and re-arrangement tool for HMTL files might solve this, but I have yet find or crete such a tool.
So the remaining candidates are the "single file for many ideas" solutions like mind maps or TiddlyWiki. Mind maps are a single hierarchy that can grow well, TiddlyWiki is more scattered with cross links, so ideal for less hierachically structured information. The MPTW variant of TiddlyWiki can do both, hierachical and cross linked.
Mind Maps are very fast in (re-)arranging, but harder to export to a sharable solution.

So the winner for the idea phase is MPTW, with Mind Maps being a close second.

2. Experiment

After incubating an idea (such as "try setting up a tabular database within DokuWiki") for enough time, I might eventually decide to give it a try - an experiment starts. When I do that, I'm usually within network range, so a blog or wiki is an option. Because it's only an experiment, possibly involving a not yet secured appication, I want to keep my notes confidential to me or very few colleagues (IT staff). The group permitted to see my notes varies by experiment, so a blog, visible by a hard administered group of people, is not (yet) adequate. A wiki with carefully chosen access permissions might be.
I want to keep information gathered in this phase, so I know later (e.g. when setting up a second or third copy of the service) which tricks worked and which didn't to get the thing going. This means, I'll want to store the results in a place that'll live longer than TiddlyWiki or mind maps.
HTML files on the intranet are too public for the experiment phase.
The sequence of actions is important, so a slightly hard to rearrange format like a wiki is o.k.

So a Wiki like DokuWiki is the winner in the experiment phase.

3. Test

After the experiment has shown in single or dual user mode, that the service can really do something useful, I usually invite a few real users (not just IT staff) to try it. "I think it works, bunt don't rely too much on it yet, tell me what to do better."

The users need a manual, FAQ and similar stuff. A disussion area would be nice too.
The IT people need a place to add many small scattered bits of experience to an already pre-arranged administration manual.

The big thing here is sharing, so only networked techniques are an option here. A blog isn't strucutured enough (just a sequence of articles), HTML files aren't flexible enough and certainly can't accomodate user input.
In my opinion, it's acceptable for offline-created contributions to be integrated later back in the office. A read only HTML copy of the wiki should be o.k. for viewing while disconnected.

So the winner for the test phase is the networked wiki.

4. Production

When the test went well, the service is put into production use for a large number of users, possibly the entire corporation. Manuals etc. should have been created in the test phase, but may need refinement now. Perhaps usage instructions and administrative manuals need a lot more detail now. Users may want to discuss about creative ways to use the service. So a lot of editing takes place, possibly for a lot of users. Everything happens on the net.
This is the time when read-only access for all users and read-write access for documentors and administrators needs to be distinguished. With that kind of massive sharing, mind maps are long lost and TiddlyWiki is too hard to edit by a group. A blog may be o.k. for announcements, but not for editing administrative procedures. Static HTML files may be an option for the user manual, but only if there are very few people editing.

Is there anything else than the wiki for the production phase?

5. Legacy

After many years in production use, eventually the demand for this kind of service diminishes. Perhaps there are many new demands, causing lots of documentation needs. Eventually a new project is started to introduce a successor service. When that is in the test phase, documentation needs increase due to transition issues. Questions like "How do I get the data out of the old system?" could be dominating the scene.
All of this happens in the network, with a lot of sharing. So again, the wiki is best for the legacy phase.

6. History

When the last user has quit using the old system and all the data is tranistioned to the new system, there is still need for documentation, especially with respect to compliance requirements. There may be laws demanding that processes (how were things supposed to be done back then?) are documented as long as 10 years. For medical information that time span could be up to 30 years after the death of the patient, which could be more than a century after the treatment of a young child.
In this phase, which could also be called "archive", nothing is going to change - in fact nothing is permitted to change, perhaps even protected by digital signatures and friends. Network access as a prerequisite to documentation access is acceptable. Publicity varies, depending on the amount of detail in the documentation, but is going to be constant.

I consinder static HTML files best for the history phase. They can be created by an HTML export from the legacy phase wiki.

Summary

In the idea phase, I consider a TiddlyWiki or perhaps a mind map as the best solution.
As soon as the experiment starts, I'd move all documentation to a DokuWiki or equivalent.
For offline use, I'd create periodic HTML exports of the DokuWiki.
For the history phase, I'd put the last HTML export on a static website, probably protected by standard web server authentication methods (e.g. .htaccess on Apache).

Mind Maps as a Documentation System?

For quite a time I like to use FreeMind as a mind mapping program. Mind maps are logical trees, arranged in a way that gives a good overview while making it easy to add details. With a good editing program, mind maps are a great tool for collecting and sorting ideas. Each mind map gets saved into a separate file, so this is similar to TiddlyWiki.

Collecting and sorting ideas/thoughts is much of what documentation is about, so it worth analyzing, to what extent mind maps in general and FreeMind in particular fulfil the requirements of the ideal documentation system:

Easy addition of thoughts is the big strength of mind maps, so they definitely can do it.
Useage on the road is o.k, because the mind map is stored in a single file without depending on a server.
Searches are well possible within a single mind map, but hard across separate mind map files. So I'd rather create a few large maps than many little maps.
Synchronization by USB pen drive is well possible because the maps are single files.
Confidentiality is possible by keeping the map confidential.
Longevity is somewhat hard, because the map is stored in a (more or less) proprietary format, although it can be exported into various formats, including long living HTML. When the FreeMind projects decides to drop it's product, chances are, I can't open my maps after 2 more major releases of Java.
Sharing with others is very hard, because I can only distribute the map file or an HTML export of it. Not precisely the way, collaboration is meant to be.

HTML files as a Documentation System?

For about 9 years now I'm using a set of linked HTML files as information storage. As an editor I use the little known Amaya. Being the testbed of the WWWconsortium, it produces the cleanest HTML I can think of.

HTML files are great in portability (synchronize the collection with your USB pen), access granularity (place less public stuff in a less public subdirectory, protected by .htaccess or similar) and longevity (HTML will probably still be readable in 20 years by then current software).
Searching can be done classic search engines like ht://Dig, although won't wok on a netbook. A desktop dearch engine might help, but I don't like Google search because of privacy concerns and Windows search because of it's power requirements.

What I feel making them hard to use is the relatively large effort required to add a new page. I need to create a new page, create a link to the parent page, edit the parent page, create a link to the child page and only then I can start writing down my quick note. To me, that kind of effort is acceptable when adding documentation about a stabilized component to the corporate intranet, but not for 3 lines about a twist that might or might not work.

After 9 years of continous information adding, some stuff needs to be reorganized. But with HTML files, this is inadequately hard, at least without the right tools, none of which I found up to now. Perhaps I'll program some of them sometime. If I just want ro rename the file name of a single page, I need to find all pages that have a link to the to-be-renamed page and change the link in every single one of them. If I want to move 15 pages into a different subdirectory, that's more of a project than a quick maintenance task. This needs tools.

Does anyone of you readers have any idea where to search for tools?

Use TiddlyWiki for Documentation?

Early this year, I discovered TiddlyWiki, a single file personal wiki. It's all in one HTML file loaded with lots of JavaScript code. Because it's open source, there are several variants. The ones I like best are MPTW (can create automatic cross references and hierarchies by smart use of tagging) and MonkeyGTD (a Getting Things Done system based on MPTW). I wrote about it already in my tech-blog.

I feel it great for taking structured notes, but a bit hard to maintain for larger (> 100 pages) collections of information.
Using it on the road is easy - just open the HTML file in Firefox or another supported browser.
It's great for synchronising between computers using a USB pen drive (just place and edit the single HTML file in your synchronized folder).
Finding stuff is easy too - it has a built in search facility.
In MPTW it's easy to create a new note by opening the parent node (you can show a multi level list of child nodes) and click on "new here".
It is easy to keep the data confidential, because the you just dont share that single big HTML file.

Sharing the entire collection read only is easy: put the HTML file on a static web server. On a file server, or on a specially prepared web server like tiddlyspot.com, there would even be read-write capability, but always for the entire collection.
Sharing only parts of the information with others is hard, because it's hard to split a single file. In theory TiddlyWiki has a mechanism to synchronize single tiddlers (the equivalent of the pages of a "classic" wiki) with a web server, but I never got that working. Perhaps I did not try hard enough.

So TiddlyWiki is great for small scale personal note taking, but not for information that's supoosed to be available for others to read (and perhaps even edit).

Use (Doku)Wiki as a Documentation System?

By a wiki, I mean a web based piece of software, installed in an intranet for privacy. I have tried DokuWiki. It is a great piece of software. I can add information really quick (wiki is said to mean quick), I can tag it and search for it. Links are created automatically when I use WikiWords in CamelCaps or put the word/phrase in [[double square brackets]].
DokuWiki (and possibly other wikis) offer a good deal of access control; enough to make sure that configuration details are read only by the IT staff. It's probably not safe enough for passwords, but there are solutions for those too, e.g. KeePass.

Hard to use while disconnected
Because a wiki is on the web (with a few exceptions, e.g. TiddlyWiki), it's by definition hard to use in disconnected mode. But I see a few options (none of them tried yet) to get around this limitation:

1. Create an HTML copy
Before I leave the office, I could use wget or a similar tool to rip the entire wiki into a set of HTML files. I could copy these files on my netbook and have robust read-only access on the road. Unfortunately, such a scheme can't have direct write access, so I'd have to write new stuff to a separate file and integrate it later when back at the office. Not nice, but workable.

2. Run a Wiki Server on the Netbook plus Synchronisation Plugin
I didn't really expect this to be possible, but there are synchronisation plugins for DokuWiki. On a quick search this morning, I found sync and freesync. Development of both has begun as late as February 2009, so this is really new.
Running a full Apache webserver on a netbook may be stretching it, but it should be possible. Perhaps I'll try it some distant day and report on my technical blog.

One limitation of this scheme is the need to synchronize over the network. This is a show stopper in my typical use case: I leave the office in a hurry, just synchronising to the USB pen. I don't have the time to boot the netbook, connect it to the network, start the web server, start firefox, move to the synchronisation page, trigger the synchronisazion, wait for the synchronisation to finish, shutdown the netbook, pack the netbook. In short: when I leave the office (network), the netbook will not be in synchronized state.
The next morning I'm on the train, outside of network connectivity, so I can't synchronize and the netbook will stay out of sync. But I'm going to produce a lot of ideas, to do items, clarifications etc. So the already out-of-sync netbook copy will be modified further. When I return to the office, there is a lot to synchronize and chances are high that there will be conflicts. Resolving synchronisation conflicts is probably harder than integrating separate notes, so I'll better create those separate notes.

3. Synchronize the raw data files through the USB pen
DokuWiki stores its pages in simple plain text files, not in a database. I could synchronize the text file using unison or a similar tool, but doing so bidirectionally, and that's required when I want to be able to edit on the road, would reqire write access to a network share of internal wiki files. I could arrange for that acces for me, because I'm the administrator, but that's not feasible for the average user.

Use a Blog as a Documentation System?

At first glance, the idea is appealing. I can create a blog post very easy, not much sorting effort. Finding stuff back is done by search options, enhanced by the tagging (label) logic. It might even work on the road, e.g. with the Gears support of wordpress.com.
But a blog is too open. Everybody in the world can read everything. If there is a privacy option, it is too coarse, like "invited members only, for every post".
So in my opinion, a blog is good for general ideas, like the one you are reading just now, but not suitable for specific internal documentation.

What I Want in my Perfect Documentation System

Add thouths quick and easy.
When I set up something or discover an interesting idea on the web, then I want to be able to do add a quick note into the system, without having to set up lots of links. I want to keep thinking on the matter at hand, not on the documentation system.

Use it on the road,
even when disconnected. Mobile phone coverage is expensive (I dont want to pay 35 Euros per month just for that) and patchy, especially in the situation where I have most time to think about things: during multi-hour train rides.

Enable me to find stuff again.
That's what a documentation system is there for. If I can't find the thoughts I entered half a year or just a week ago, I could throw the ideas away in the first place an stop bothering with documentation.

Synchronize it among mutilple computers using a USB pen drive.
I want to be able to edit the documentation on my netbook (small and light on the road), my home computer (large screen) and my office computer.
scas

Keep documentation confidential.
I don't want everybody in the world to know about the configuration of VPNs etc. Not even everybody in my office needs to know, how I secured the experimental wiki. So a publicly readable wiki is too open for me.

Be able to retrieve the information in ten or twenty years,
when the software I used for writing it, has become obsolete. So the storage needs to be in a format readable by anything. This requirement forbids proprietary binary formats. Simple text or HTML files are ideal; a method to rip the entire system into a set of HTML files would be an acceptable substitute.

Share parts with others,
file by file, or category by category. The organisation I work for has branches. The IT guy at branch A needs to know about the setup in branch A, but not all of my vague, incomplete ideas. My colleague in the office needs to know the setup of many office things, but not the SSID of my home WLAN.