Getting Boxes Done, the Code

Benjamin Franklin once wrote,

“For every minute spent in organizing, an hour is earned.”

Due to Emacs, I may have accrued nearly a lifetime by now.

As I mentioned in the first part of this essay, my shiny idea syndrome filled my task list with half-baked ideas, shopping lists, books to read, and solutions to all the world’s problems.

Assuming you like the idea of storing your project ideas in Org, the rest of essay describes the details and the code I use. Since the code for this project is tangled from this essay, ala literate programming, this page should stay up-to-date. ;-)

Workflow with an Example

As I mentioned before, I pretend ideas are on a virtual index card, and process them as I move them around organizational boxes. After creating a flowchart, I simplified it into more of a workflow:


What are the red letters, you ask? Those are the key bindings I use to move the virtual index cards from my inbox to other boxes (files or directories). You’ll see where I specify these and their implementation later.

Perhaps an example of using this workflow would be helpful.

I may notice that I’m running low on personal supplies, and quickly jot a note in Orgzly:


This puts a line in my Inbox:

* Buy toothpaste and eye drops

The next morning, as I’m reviewing the entries, I hit t to refile (move) the entry into my Task List. After I’ve been shopping, and later review the task list, I hit a to archive it, or c to archive it the completed file, which for me, is that day’s journal entry (however, you could specify a file like, Completed tasks in org are typically archived and I discuss the details of this later on.

How about another example? The other day while noticing the large number of uncommitted text files (why yes, my org directories are under git control), I thought of an idea. I quickly kick off an Org capture and put another entry in my Inbox:

* Can I do a git commit automatically (at least, more regularly)?

When reviewing my Inbox the next day, I realize, while a good idea, I have no clue how to accomplish it. Since it isn’t ready, I refile it to my incubate file (by typing i). Returning to my incubating entries I hatch a plan (which in this example involves a little web searching). With a plan, the wish becomes a project, I update the entry:

* TODO Git commit on Sleep
  SCHEDULED: <2018-12-28 Fri>

  Certain directories under git-control often go weeks or months between
  commits, as "projects" containing text notes don't really have a *change* to
  commit. What about a checkpoint? Perhaps closing laptop or system sleeps, we
  commit all files.

  * On a Mac, checkout [[][SleepWatcher]]
  * On Linux, checkout [[][pm-utils]]

  Goal should be to have a script that both programs can run, e.g.

  #+BEGIN_SRC shell
      ~/bin/sleepwatcher -d -s ~/bin/on-sleep

Note the TODO label and scheduled date, as these make it show on my agenda.

As a real project, I refile it from incubate to tasks using t, or if it’s a larger, involved project, to its own file in projects using m P. Completion of a project moves it again, but since this project contains notes I may have to reference later, I shuffle the entire file to my technical directory.

Why so many steps and so many files? In a large project management system, tasks are typically associated with states in a database where views present tasks in different states. I’m working with text files, so tasks in different states live in different files. Normally, this would be painful, but not with org-mode. Single key sequences shuffle the text to different files, allowing me to view state by simply viewing a file.

If this concept is clear, the remaining sections include my procedures (and Emacs functions) for processing ideas from inception to completion using the examples described above.

See all those destinations in the workflow illustration above? Let’s create constant variables, where the functions can use them later:

(defvar org-default-projects-dir   "~/projects"                     "Primary GTD directory")
(defvar org-default-technical-dir  "~/technical"                    "Directory of shareable notes")
(defvar org-default-personal-dir   "~/personal"                     "Directory of un-shareable, personal notes")
(defvar org-default-completed-dir  "~/projects/trophies"            "Directory of completed project files")
(defvar org-default-inbox-file     "~/projects/"         "New stuff collects in this file")
(defvar org-default-tasks-file     "~/projects/"           "Tasks, TODOs and little projects")
(defvar org-default-incubate-file  "~/projects/"        "Ideas simmering on back burner")
(defvar org-default-completed-file nil                              "Ideas simmering on back burner")
(defvar org-default-notes-file     "~/personal/"   "Non-actionable, personal notes")
(defvar org-default-media-file     "~/projects/"           "White papers and links to other things to check out")

Why yes, if you steal my code, you’ll want to change those values.

Oh, and since some of these are pretty cool and could have items that would show up on an agenda, we may want to add them to org-agenda-files automatically:

(add-to-list 'org-agenda-files org-default-inbox-file)
(add-to-list 'org-agenda-files org-default-tasks-file)

Or, you can use code like this to add all files in the projects directory as potential agenda files:

(setq org-agenda-files (list org-default-projects-dir))
(setq org-agenda-file-regexp "^[a-z0-9-_]")

However, let’s begin by keeping it simple:

(dolist (agenda-file (list org-default-inbox-file
  (add-to-list 'org-agenda-files (expand-file-name agenda-file)))

The Inbox,

Everything begins as an entry in the Inbox (org-default-inbox-file) that I call (seemed like a good filename to avoid feeling overwhelmed), so I need a flexible capturing entry to make it easy to create entries in that file:

(add-to-list 'org-capture-templates
             `("t" "Task Entry"        entry
               (file ,org-default-inbox-file)
               "* %?\n:PROPERTIES:\n:CREATED:%U\n:END:\n\n%i\n\nFrom: %a"
               :empty-lines 1))

Calling org-capture (which most people bind to C-c c, but Spacemacs has as SPC C c) might look like the following screenshot:


Note: With Orgzly (or other mobile org-specific application) configured to use this file as its default destination for all new ideas or todos, your mobile device will match your system.

Tidying by Refiling

As a temporary holding spot for all incoming tasks, notes and ideas, the goal is to keep the Inbox empty. Every day, I review the collected entries, and adjust and move them to appropriate locations.

I have created a little helper system for processing these ideas through the workflow shown in the flowchart above. Here is a screenshot of it in action where I am currently determining what to do with an entry captured earlier. The menu at the bottom of the screen matches my workflow shown earlier:


Note: The screenshot above may not match the code you’ll see below. Since this essay is really just a literate programming approach to generating the code I actually use, I’ll be changing the code, but may not re-render the images. Sorry.

While Emacs has many options for calling functions, I’ve chosen to use the Hydra project to present a focused collection of keybindings to functions. Originally I made a minor mode, but found Hydra’s hint screen and repetitive key calls easier (once I start the hydra, I can simply type i multiple times to refile multiple entries to the incubation chamber).

Here is the definition for the Hydra, and you’ll notice that after the hint display, it is just a list of keybindings to functions:

(defhydra hydra-org-refiler (org-mode-map "C-c s" :hint nil)
  ^Navigate^      ^Refile^       ^Move^           ^Update^        ^Go To^        ^Dired^
  _k_: ↑ previous _t_:   tasks     _m X_: projects  _T_: todo task  _g t_: tasks    _g X_: projects
  _j_: ↓ next     _i_/_I_: incubate  _m P_: personal  _S_: schedule   _g i_: incubate _g P_: personal
  _c_: archive    _p_:   personal  _m T_: technical _D_: deadline   _g x_: inbox    _g T_: technical
  _d_: delete     _r_:   refile                   _R_: rename     _g n_: notes    _g C_: completed
    ("<up>" org-previous-visible-heading)
    ("<down>" org-next-visible-heading)
    ("k" org-previous-visible-heading)
    ("j" org-next-visible-heading)
    ("c" org-archive-subtree-as-completed)
    ("d" org-cut-subtree)
    ("t" org-refile-to-task)
    ("i" org-refile-to-incubate)
    ("I" org-refile-to-another-incubator)
    ("p" org-refile-to-personal-notes)
    ("r" org-refile)
    ("m X" org-refile-to-projects-dir)
    ("m P" org-refile-to-personal-dir)
    ("m T" org-refile-to-technical-dir)
    ("T" org-todo)
    ("S" org-schedule)
    ("D" org-deadline)
    ("R" org-rename-header)
    ("g t" (find-file-other-window org-default-tasks-file))
    ("g i" (find-file-other-window org-default-incubate-file))
    ("g x" (find-file-other-window org-default-inbox-file))
    ("g c" (find-file-other-window org-default-completed-file))
    ("g n" (find-file-other-window org-default-notes-file))
    ("g X" (dired org-default-projects-dir))
    ("g P" (dired org-default-personal-dir))
    ("g T" (dired org-default-technical-dir))
    ("g C" (dired org-default-completed-dir))
    ("[\t]" (org-cycle))
    ("s" (org-save-all-org-buffers) "save")
    ("<tab>" (org-cycle) "toggle")
    ("q" nil "quit"))

We just need to bind something like C-c s to call hydra-org-refiler/body. Or, under Spacemacs with a leader key:

(spacemacs/set-leader-keys-for-major-mode 'org-mode "r" 'hydra-org-refiler/body)

Or Doom’s leader key:

(map! :map org-mode-map :localleader
      :desc "boxes" "B" 'hydra-org-refiler/body)

Now I have a system for calling all these mover and shaker functions. Some functions, like org-schedule, come with Org, while others, I’ll need to write.

Ideas in Incubation:

Things and ideas that need fleshing before tackling go into an incubation file. Org has a nice feature for moving a subtree (an Org header entry and its contents) to another file, org-refile (see the Org Manual for details).

Calling the org-refile function presents a list of target files, which you need to program using an odd cons entry. I found specifying these easier to learn by example. Along with the target file, you need to specify the heading levels presented as a destination. Since I want top-level entries from Inbox to go to top-level entries in the Incubation file, I specify a level of 0:

(setq org-refile-targets `((,org-default-incubate-file :level . 0)))

Not familiar with that back-tick/comma syntax? You can define a list of symbols with '(a b c), where a is not a variable, but just a Lisp symbol. But what if you wanted the b to actually refer to the value of a variable? For this, you replace the normal tick ' (apostophe or single quote) with a back-tick, ` (or back quote), which acts the same, except that anything that follows a comma is a variable, and Emacs replaces it with its value. I guess I could have written that code as:

(setq org-refile-targets (list (list org-default-incubate-file :level . 0)))

But now I am getting off topic… sorry for that interlude.

Anyway, to allow refiling entries to as a top-level destination, we need to make two variable changes:

(setq org-refile-use-outline-path 'file
      org-refile-allow-creating-parent-nodes t
      org-outline-path-complete-in-steps nil)

Now, move to an org-mode section, and call org-refile (with either C-c C-w or , s r if you are using Spacemacs or SPC m r r in Doom … wait, I shouldn’t bother with all the keystrokes, as you, gentle reader, can look that up with C-h f or SPC h f if you are using Spacemacs or Doom… and here I did it again).

Notice that shows this as a destination.

Since I last updated this essay, I realize that I incubate a lot of ideas, and I figured I might want to expand the incubating refiling to any file that starts with incubate-

(defun boxes--get-incubators ()
  "Calls `completing-read' with a multiple choice of incubating files.
That is, any files located in the `org-default-projects-dir' directory
that begin with `incubate-'."
  (let ((choices (cons (cons "incubator" org-default-incubate-file)
                       (--map (cons (replace-regexp-in-string ".*incubate-\\(.*\\).org" "\\1" it)  it)
                              (directory-files org-default-projects-dir t "incubate-.*.org")))))
    (assoc (completing-read "Refile to Incubator:" choices) choices 'equal)))

(defun org-refile-to-another-incubator (incubator)
  "Refile the current org subtree to an incubate file.
A list of available `incubate-*.org' files are presented as a choice."
  (interactive (list #'boxes--get-incubators))
  (org-refile nil nil (list (car incubator) (cdr incubator) nil 0)))

Well-developed tasks go into the file (which will show on my agenda):

(setq org-refile-targets `((,org-default-incubate-file :level . 0)
                           (,org-default-tasks-file :level . 0)))

Books to read and music to check out go into appropriate sub-sections in my file, so I set :level to 1 for this target destination:

(setq org-refile-targets `((,org-default-incubate-file :level . 0)
                           (,org-default-media-file :level . 1)
                           (,org-default-tasks-file :level . 0)))

Now, refiling should look something like:

Refile subtree to:

However, every file in my projects directory could be a potential destination, and with the hydra making typical destinations easy, I could specify them all:

(setq org-refile-targets
      (append `((,(expand-file-name org-default-media-file) :level . 1)
                (,(expand-file-name org-default-notes-file) :level . 0))
               (directory-files org-default-projects-dir nil ".org")
               (-remove-item (file-name-base org-default-media-file))
               (--remove (s-starts-with? "." it))
               (--remove (s-ends-with? "_archive" it))
               (--map (format "%s/%s" (expand-file-name org-default-projects-dir) it))
               (--map `(,it :level . 0)))))

(setq org-refile-target-table nil)

I am taking advantage of a threading macro from Magmar’s lovely dash library. In case you are unfamiliar with the ->> macro, this code gets a list of all the Org files, then removes the media file and any hidden files. It then takes that list of files and creates a new list of fully-qualified file names and appends the :level of 0. Once you get the idea of the threading macro, the code seems much easier to read.

If you haven’t installed dash, require the subr-x library, and call the more verbose thread-last macro, as it does the same thing.


Refiling Programmatically

I can’t call org-refile programmatically with a file destination (as needed in the hydra shown above), as it is only interactive. More problematic, org-refile is a monolithic function, so I can’t call any helper functions it might use. I currently see no other approach but to implement my own simpler re-filer.

Copying regions is what Emacs does well…so, let’s define a region of a subtree:

(defun org-subtree-region ()
  "Return a list of the start and end of a subtree."
    (list (progn (org-back-to-heading) (point))
          (progn (org-end-of-subtree)  (point)))))

Now we can use that function to kill the region of a subtree, open a file (the target destination), and insert the previous contents:

(defun org-refile-directly (file-dest)
  "Move the current subtree to the end of FILE-DEST.
If SHOW-AFTER is non-nil, show the destination window,
otherwise, this destination buffer is not shown."
  (interactive "fDestination: ")

  (defun dump-it (file contents)
    (find-file-other-window file-dest)
    (goto-char (point-max))
    (insert "\n" contents))

    (let* ((region (org-subtree-region))
           (contents (buffer-substring (first region) (second region))))
      (apply 'kill-region region)
      (if org-refile-directly-show-after
          (save-current-buffer (dump-it file-dest contents))
        (save-window-excursion (dump-it file-dest contents))))))

(defvar org-refile-directly-show-after nil
  "When refiling directly (using the `org-refile-directly'
function), show the destination buffer afterwards if this is set
to `t', otherwise, just do everything in the background.")

After moving a subtree, do I want to see the resulting buffer in a window? If so, I use save-current-buffer (shown above), otherwise, this function calls save-window-execursion. I haven’t decided which I like best, so I have a customization variable that I can change.

Now, let’s create functions for the most-used refile destinations used by the Hydra (notice that the Hydra can also call org-refile directly in order to get access to all targets):

(defun org-refile-to-incubate ()
  "Refile (move) the current Org subtree to `org-default-incubate-fire'."
  (org-refile-directly org-default-incubate-file))

(defun org-refile-to-task ()
  "Refile (move) the current Org subtree to `org-default-tasks-file'."
  (org-refile-directly org-default-tasks-file))

(defun org-refile-to-personal-notes ()
  "Refile (move) the current Org subtree to `org-default-notes-file'."
  (org-refile-directly org-default-notes-file))

(defun org-refile-to-completed ()
  "Refile (move) the current Org subtree to `org-default-completed-file',
unless it doesn't exist, in which case, refile to today's journal entry."
  (if (and org-default-completed-file (file-exists-p org-default-completed-file))
      (org-refile-directly org-default-completed-file)
    (org-refile-directly (get-journal-file-today))))

Scheduling and Planning

While reviewing the collected ideas in my Inbox, I often need to tidy them before moving them around. Add a TODO label to each task with T in my hydra, as well as schedule a date with an S (as a task without due date is just a wish). Before I move the subtree, I may need to change the header’s text (which I added to the hydra with an R key):

(defun org-rename-header (label)
  "Rename the current section's header to LABEL, and moves the
point to the end of the line."
  (interactive (list
                (read-string "Header: "
                             (substring-no-properties (org-get-heading t t t t)))))
  (replace-string (org-get-heading t t t t) label))

Completing Tasks

Completed tasks are typically archived, using org-archive-subtree. This is a brilliant feature for keeping your notes relevant (as long as this feature doesn’t surprise you by an accidental fat-fingering of the keyboard):


To archive to a separate file like this, set the variable, org-archive-location to "%s_archive".

However, in my world tasks marked as completed originate from multiple file locations; I would rather move these finished tasks to my journal. I use org-journal with a file for each day; for instance, 20180113 (without the .org extension…see my journal helper functions if you are curious).

This function archives a subtree to today’s journal entry (marking the task completed in the process):

(defun org-archive-subtree-as-completed ()
  "Archives the current subtree to today's current journal entry."
  ;; According to the docs for `org-archive-subtree', the state should be
  ;; automatically marked as DONE, but I don't notice it, so let's force:
  (when (not (equal "DONE" (org-get-todo-state)))
    (org-todo "DONE"))

  (let* ((org-archive-file (or org-default-completed-file
         (org-archive-location (format "%s::" org-archive-file)))

The following function returns the filename of today’s journal entry (which I can use as a refile destination):

(defun todays-journal-entry ()
  "Return the full pathname to the day's journal entry file.
Granted, this assumes each journal's file entry to be formatted
with year/month/day, as in `20190104' for January 4th.

Note: `org-journal-dir' variable must be set to the directory
where all good journal entries live, e.g. ~/journal."
  (let* ((daily-name   (format-time-string "%Y%m%d"))
         (file-name    (concat org-journal-dir daily-name)))
    (expand-file-name file-name)))

Refiling to Org Files

Archiving completed org entries (subtrees) is common, but sometimes a project entry grows and deserves its own file. While Org can refile or archive a subtree to a new file, these functions simply create a new file and copy the subtree. Personally, I really want a subtree to become a proper org file.

Writing that code became large and perhaps a bit of a white rabbit hole for most readers of this essay. Interested? Please, check out Part 3 of this essay.


We now have a Hydra that easily kicks off functions that help keep my task inbox manageable. Each morning, I would like to start an environment when my Inbox is loaded and ready for refiling. The following function does that by calling functions I would normally do manually:

(defun org-boxes-workflow ()
  "Load the default tasks file and start our hydra on the first task shown."
  (let ((org-startup-folded nil))
    (find-file org-default-inbox-file)
    (pop-to-buffer (get-file-buffer org-default-inbox-file))
    (goto-char (point-min))
    (org-next-visible-heading 1)

This assumes that I have a special agenda display:

(defun ha/org-agenda ()
  "Displays my favorite agenda perspective."
  (org-agenda nil "a")
  (get-buffer "*Org Agenda*")
  (execute-kbd-macro (kbd "A t")))

I hope you can grab and modify this code to suit your own needs and whims. I may attempt to make the code more general and useful to others, but at the moment, I’d like to see if this code has any legs.

Let me know if you have any improvements you’d make.

Date: 2018-12-24 December

Created: 2023-01-11 Wed 21:34