www.howardism.org
Babblings of an aging geek in love with the Absurd, his family, and his own hubris.... oh, and Lisp.

Getting Boxes Done, the Code

Benjamin Franklin once wrote, “For every minute spent in organizing, an hour is earned.” Due to Emacs, I think I have accrued nearly a lifetime by now.

As I mentioned in the first part of this essay, my shiny idea syndrome filled my task list with half-baked ideas, shopping lists, books to read, and solutions to all the world’s problems.

Assuming you like the idea of storing your project ideas in Org, the rest of essay describes the details and the code I use. Since the code for this project is tangled from this essay, ala literate programming, this page should stay up-to-date. ;-)

Workflow with an Example

As I mentioned before, I pretend ideas are on a virtual index card, and process them as I move them around my organizational boxes. After creating a flowchart, I simplified it into more of a workflow:

getting-more-boxes-done-01.jpg

Perhaps an example of using this workflow would be helpful.

My captured entries begin life with a single line in the Inbox, breathe.org, for example:

* Can I do a git commit automatically (at least, more regularly)?

Which I then refiled to my incubate file for fleshing out. Returning to the entry to hatch a plan (which often involves a little web searching). With a plan, the wish becomes a project:

* TODO Git commit on Sleep
  SCHEDULED: <2018-12-28 Fri>

  Certain directories under git-control often go weeks or months between
  commits, as "projects" containing text notes don't really have a *change* to
  commit. What about a checkpoint? Perhaps closing laptop or system sleeps, we
  commit all files.

  * On a Mac, checkout [[https://www.bernhard-baehr.de/][SleepWatcher]]
  * On Linux, checkout [[https://launchpad.net/ubuntu/+source/pm-utils][pm-utils]]

  Goal should be to have a script that both programs can run, e.g.

  #+BEGIN_SRC shell
      ~/bin/sleepwatcher -d -s ~/bin/on-sleep
  #+END_SRC

Note the TODO label and scheduled date, as these make it show up on my agenda.

As a real project, I move it from incubate to either tasks, or (if it is a large, involved project), its own file in projects. Completion of a project moves it again, to either the trophies directory, or if it has notes worth remember, I shuffle it to the technical directory for safe-keeping.

Why so many steps and so many files? In a large project management system, tasks are typically associated with states in a database where views present tasks in different states. I’m working with text files, so tasks in different states live in different files. Normally, this would be painful, but not with org-mode.

If this concept is clear, the remaining sections include my procedures (and Emacs functions) for processing my ideas from inception to completion.

See all those destinations in the workflow? Let’s create constant variables, where the functions can use them:

(defvar org-default-projects-dir  "~/projects"                   "Primary GTD directory")
(defvar org-default-technical-dir "~/technical"                  "Directory of shareable notes")
(defvar org-default-personal-dir  "~/personal"                   "Directory of un-shareable, personal notes")
(defvar org-default-inbox-file    "~/projects/breathe.org"       "New stuff collects in this file")
(defvar org-default-tasks-file    "~/projects/tasks.org"         "Tasks, TODOs and little projects")
(defvar org-default-incubate-file "~/projects/incubate.org"      "Ideas simmering on back burner")
(defvar org-default-notes-file    "~/personal/general-notes.org" "Non-actionable, personal notes")
(defvar org-default-completed-dir "~/projects/trophies"          "Completed projects")
(defvar org-default-media-file    "~/projects/media.org"         "White papers and links to other things to check out")

The Inbox, breathe.org

Everything begins as an entry in the INBOX (org-default-inbox-file) that I call breathe.org (seemed like a good filename to avoid feeling overwhelmed), so I need a flexible capturing entry:

(add-to-list 'org-capture-templates
             `("t" "Task Entry"        entry
               (file ,org-default-inbox-file)
               "* %?\n:PROPERTIES:\n:CREATED:%U\n:END:\n\n%i\n\nFrom: %a"
               :empty-lines 1))

Which might look like the following screenshot:

getting-more-boxes-done-02.png

Configure Orgzly with this file as the default destination for all new ideas or todos. Then your mobile device will match your system.

Tidying by Refiling

As a temporary holding spot for all incoming tasks, notes and ideas, the goal is to keep this file empty. Every day, I review the collected entries, and adjust and move them to appropriate locations.

I have created a little helper system for processing these ideas through the workflow shown in the flowchart above. Here is a screenshot of it in action:

getting-more-boxes-done-04.png

Note: The screenshot above won’t match the code you’ll see below. Since this essay is really just a literate programming approach to generating the code I actually use, I’ll be changing the code, but I’m not going to re-render the images. Sorry.

I need a collection of functions to easily archive or refile each subtree entry. A hydra can easily call these functions:

(defhydra hydra-org-refiler (org-mode-map "C-c s" :hint nil)
    "
  ^Navigate^      ^Refile^            ^Move^           ^Update^        ^Go To^        ^Dired^
  ^^^^^^^^^^-------------------------------------------------------------------------------------
  _k_: ↑ previous _r i_: incubate  _r x_: projects  _T_: todo task  _g b_: Inbox    _p x_: Projects
  _j_: ↓ next     _r j_: journal   _r p_: personal  _S_: schedule   _g t_: Tasks    _p p_: Personal
  ^ ^    heading  _r n_: task      _r t_: technical _D_: deadline   _g i_: Incubate _p t_: Technical
  _d_: delete     _r r_: refile other             _R_: rename     _g n_: Notes
  "
    ("<up>" org-previous-visible-heading)
    ("<down>" org-next-visible-heading)
    ("k" org-previous-visible-heading)
    ("j" org-next-visible-heading)
    ("g b" (find-file org-default-inbox-file))
    ("g t" (find-file org-default-tasks-file))
    ("g i" (find-file org-default-incubate-file))
    ("g n" (find-file org-default-notes-file))
    ("r i" org-refile-to-incubate)
    ("r j" org-refile-to-journal)
    ("r n" org-refile-to-task)
    ("r r" org-refile)
    ("r x" org-refile-to-projects)
    ("r p" org-refile-to-personal)
    ("r t" org-refile-to-technical)
    ("d" org-kill-subtree)
    ("S" org-schedule)
    ("D" org-deadline)
    ("T" org-todo)
    ("R" org-rename-header)
    ("p x" (dired org-default-projects-dir))
    ("p p" (dired org-default-personal-dir))
    ("p t" (dired org-default-technical-dir))
    ("q" nil "quit"))

We just need to bind something like C-c s to call hydra-org-refiler/body.

Or, under Spacemacs with a leader key:

(spacemacs/set-leader-keys-for-major-mode 'org-mode "r" 'hydra-org-refiler/body)

Note: Use Orgmode’s refiling feature for shuffling a header and its text (called a subtree) from one file to another. While I begin discussing getting stuff out of my initial collection inbox, breathe.org, the following section shows all the code for moving subtrees around.

Inbox: incubate.org

This file in the projects directory holds things and ideas that need fleshing… basically, good ideas and potential projects. This should be a refile destination:

(setq org-refile-targets '((org-default-incubate-file :level . 0)))

To allow entries (org-mode sections) called with org-refile to have incubate.org as a top-level destination, however, we need to make two changes:

(setq org-refile-use-outline-path 'file
      org-outline-path-complete-in-steps nil)

To move an org-mode section, move the point (cursor) into it, and call org-refile (with either C-c C-w or , s r if you are using Spacemacs), and notice that incubate.org shows as a destination.

tasks.org

Add any other destinations you might find helpful. Well thought tasks go into the tasks.org (which will show on my agenda):

(setq org-refile-targets '((org-default-incubate-file :level . 0)
                           (org-default-tasks-file :level . 0)))

Books to read and music to check out go into appropriate sections in my media.org file, so I set :maxlevel to 1 for that destination:

(setq org-refile-targets '((org-default-incubate-file :level . 0)
                           (org-default-media-file :maxlevel . 1)
                           (org-default-tasks-file :level . 0)))

Completed Tasks go to the Journal

Completed tasks are usually archived, using org-archive-subtree, however, I would rather move completed tasks and random notes to my journal. This function returns the filename of today’s journal entry to be used as a refile destination:

(defun todays-journal-entry ()
  "Return the full pathname to the day's journal entry file.
Granted, this assumes each journal's file entry to be formatted
with year/month/day, as in `20190104' for January 4th.

Note: `org-journal-dir' variable must be set to the directory
where all good journal entries live, e.g. ~/journal."
  (let* ((daily-name   (format-time-string "%Y%m%d"))
         (file-name    (concat org-journal-dir daily-name))
         (journal-file (expand-file-name file-name))))
  (list journal-file))

No, I’m not sure why functions used as targets for org-refile need to return a list. Anyway, here is the final list of all variable values:

(setq org-refile-use-outline-path 'file
      org-indent-indentation-per-level nil
      org-outline-path-complete-in-steps nil
      org-refile-targets (append '((org-default-media-file :maxlevel . 1)
                                   ;; Note: Any items listed here need to be removed below
                                   ;; Completed tasks go into today's journal entry:
                                   (todays-journal-entry :level . 0))
                                 (->>
                                  (directory-files org-default-projects-dir nil ".org")
                                  (-remove-item (file-name-base org-default-media-file))
                                  (--remove (s-starts-with? "." it))
                                  (--map (format "%s/%s" org-default-projects-dir it))
                                  (--map `(,it :level . 0)))))

Wait, what happened there, Howard? After adding the media with a level of 1, and the today’s journal, I then added every file in the project directory by taking advantage of Magmar’s lovely dash library.

Now calling org-refile shows me a goodly list of destinations for entries casually collected:

getting-more-boxes-done-03.png

Refiling Directly

Refiling destinations refer to existing files, but the hydra shown above needs to be able to refile to directories as well. Also, I can’t call org-refile directly with a destination; it is only interactive. More problematic, org-refile is a monolithic function.

I currently see no other approach, but to implement my own simpler refiler. Copying regions is what Emacs does well…so, let’s define a region of a subtree:

(defun org-subtree-region ()
  "Return a list of the start and end of a subtree."
  (save-excursion
    (list (progn (org-back-to-heading) (point))
          (progn (org-end-of-subtree)  (point)))))

Now kill that region, open a file destination, and insert it:

(defun org-refile-directly (file-dest &optional show-after)
  "Move the current subtree to the end of FILE-DEST.
If SHOW-AFTER is non-nil, the destination window will be shown,
otherwise, this destination buffer is not shown."
  (interactive "fDestination: ")

  (defun dump-it (file contents)
    (find-file-other-window file-dest)
    (goto-char (point-max))
    (insert "\n" contents))

  (save-excursion
    (let* ((region (org-subtree-region))
           (contents (buffer-substring (first region) (second region))))
      (apply 'kill-region region)
      (if show-after
          (save-current-buffer (dump-it file-dest contents))
        (save-window-excursion (dump-it file-dest contents))))))

After moving a subtree, do I want to see the resulting buffer in a window? If so, then I use save-current-buffer, otherwise, call save-window-execursion.

Now, I can create functions for the most used refile destinations:

(defun org-refile-to-incubate ()
  (interactive)
  (org-refile-directly org-default-incubate-file))

(defun org-refile-to-task ()
  (interactive)
  (org-refile-directly org-default-tasks-file))

(defun org-refile-to-journal ()
  (interactive)
  (org-refile-directly (get-journal-file-today)))

Scheduling and Planning

Add a TODO label to each task (with a T in my hydra) as well as schedule a date (with an S), as a task without due date is just a wish. Before I move the subtree, I may need other touchup, like easily changing the header (which I added to the hydra with a R key):

(defun org-rename-header (label)
  "Rename the current section's header to LABEL, and moves the
point to the end of the line."
  (interactive (list
                (read-string "Header: "
                             (substring-no-properties (org-get-heading t t t t)))))
  (org-back-to-heading)
  (replace-string (org-get-heading t t t t) label))

Deleting Entries

Not everything I store needs to be remember or acted. Some, like my shopping list, I delete after completing. Not sure why we don’t have a simple function to delete an org subtree section, but seems easy enough:

(defun org-kill-subtree ()
  "Deletes an org section (subtree) including the header and all
content it 'contains'.  Move the content to the kill-ring."
  (interactive)
  (let ((region (org-subtree-region)))
    (kill-region (first region) (second region))
    (org-next-visible-heading 1)))

Tidying to Files

Shuffling org entries (subtrees) from one file to another is common, but sometimes an entry grows and deserves its own file. Org doesn’t have this use case, so let’s make it.

First, I will need a function to collect the subtree’s goodies like the contents of the header and its body:

(defun org-subtree-metadata ()
  "Return a list of key aspects of an org-subtree. Includes the
following: header text, body contents, list of tags, region list
of the start and end of the subtree."
  (save-excursion
    (let* ((header     (substring-no-properties (org-get-heading t t t t)))
           (tags       (progn (org-back-to-heading) (org-get-tags)))
           (del-tags   (org-set-tags-to ""))
           (head-start (point))
           (body-start (progn (forward-line) (point)))
           (body-end   (progn (org-end-of-subtree nil t)))
           (body       (buffer-substring-no-properties body-start body-end)))
      (list header body tags (list head-start body-end)))))

I need to choose a good filename to place the contents. Sure, I would like to take the section’s header, however, what should I do with something like, Let's go Shopping! … in this case, I’ll convert all the non-alphanumeric characters to dashes and lowercase everything:

(defun org-filename-from-title (title)
  "Creates a useful filename based on a header string, TITLE.
For instance, given the string:    What's all this then?
     This function will return:    whats-all-this-then"
  (let* ((no-letters (rx (one-or-more (not alphanumeric))))
         (init-try (->> title
                        downcase
                        (replace-regexp-in-string "'" "")
                        (replace-regexp-in-string no-letters "-"))))
    (string-trim init-try "-+" "-+")))

The function org-refile-subtree-to-file is the primary entry into this concept. It takes a directory destination and moves the section to a new file there.

(defun org-refile-subtree-to-file (dir)
  "Archive the org-mode subtree and create an entry in the
directory folder specified by DIR. It attempts to move as many of
the subtree's properties and other features to the new file."
  (interactive "DDestination: ")
  (destructuring-bind (header body tags region) (org-subtree-metadata)
    (let* ((filename  (org-filename-from-title header))
           (filepath  (format "%s/%s.org" dir filename)))
      (apply delete-region region)
      (org-archive-subtree-to-file filepath header body tags))))

The heavy lifting of the previous function is actually done by this function, which given all the information it needs, deletes the subtree, creates a new file, and updates it. Note: By updating the file, we can take advantage of directory of file-specific auto-insert.

(defun org-archive-subtree-to-file (filepath header body tags)
  "Use `org-archive-subtree' to move the current subtree to the
file given by FILEPATH. This requires pre-knowledge of subtree,
including the HEADER, BODY and any associated TAGS."
    (find-file-other-window filepath)
    (goto-char (point-min))
    ;; Insert if not found:
    (replace-regexp "^#\\+TITLE:.*" (concat "#+TITLE:  " header))
    (re-search-forward "^\s*$")
    (when tags
      ;; TODO Search for TAGS: line and replace instead of inserting
      (insert (concat "#+TAGS:    " (s-join " " tags) "\n\n")))
    (replace-regexp "^\\* 0$" "")
    (org-archive-subtree-to-file-move-props))

(defun org-archive-subtree-to-file-move-props ()
  "The `org-archive-subtree' function copies a subtree to the new
file, leaving the subtree's properties under the heading, and
these really should be moved to the top-level of the file.

It is a bit complicated and ugly, but it is an ugly affair."
  (save-excursion
    (goto-char (point-min))
    (re-search-forward "^\s*$" nil t)
    (setq property-location (point))

    ;; Copy the section's properties to the top of the file:
    (org-next-visible-heading 1)
    (cl-loop for elem in (org-entry-properties)
             do
             (destructuring-bind (key . val) elem
               (when (not (--any? (equal key it)
                                  '("ITEM" "CATEGORY" "PRIORITY"
                                    "BLOCKED" "FILE")))
                 (goto-char property-location)
                 (insert "#+PROPERTY: " key " " val "\n"))))

    ;; Delete the subtree header and its properties:
    (let ((start (progn (goto-char (point-min))
                        (re-search-forward "^$")
                        (point)))
          (end   (re-search-forward ":END:")))
      (delete-region start end))))

Using auto-insert to pre-populate a file is great, so I need a way to make sure certain lines are set correctly, and if not, insert them:

(defun org-set-file-property (property value &optional current-point)
  "Where PROPERTY is a top-level, file-wide property, like
`TITLE' or `TAGS', this function makes sure that the property
contains the contents of VALUE, and if the file doesn't have the
property, it is inserted at either the CURRENT-POINT if non-nil,
or at the next available blank line."
  (save-excursion
    (let* ((upprop   (upcase property))
           (rexpr    (format "#\\+%s: .*" upprop))
           (contents (format "#+%s: %s" upprop value))
           (start    (point))
           (end      (progn (re-search-forward "^\s*$") (point))))
      (if (re-search-forward rexpr end)
          (replace-match contents)
        (goto-char end)
        (insert contents)
        (insert "\n"))
      (message rexpr)
      )))

(org-set-file-property "foo" "bling") ;; Let's try this ... point point on this line:
;; Then we should be able to replace:
;; foo: bar
#+FOO: bar

;; However, we should not find badaboom at all.
#+FOO: bling

I think I spent a bit too much time worry about text that can be easily manipulated afterwards. Well, now I can finish my workflow.

Projects

Good, big, chewy projects may start life as a single idea in incubate but when they outgrow that file, they need to be moved to their own org file in the projects directory.

(defun org-refile-to-projects ()
  "Move the current subtree to a file in the `projects' directory."
  (interactive)
  (org-refile-subtree-to-file org-default-projects-dir))

Completed projects need to be moved out of the projects directory. This could be:

  • technical : where complicated notes can help with ongoing maintenance
  • personal : need to remember personal information about the project
  • projects/trophies : seems like an apt name for a done directory

Sounds like a job for dired, eh?

Technical Folder

The technical folder contains any notes on non-work, non-personal information. The idea with this box is that I can share it publicly.

(defun org-refile-to-technical ()
  "Move the current subtree to a file in the `technical' directory."
  (interactive)
  (org-refile-subtree-to-file org-default-technical-dir))

Personal Folder

Any thing to be remembered or referenced goes into a file in the personal folder. Each of these files end with a .txt extension so that Dropbox can display it. However, it is still an org file, so my Yasnippet template for it looks like:

--org--
#+TITLE:  $1
#+AUTHOR: Howard Abrams
#+EMAIL:  howard.abrams@gmail.com
#+TAGS:   personal $2

$0

We need an auto insert for anything in that directory to expand that snippet.

(define-auto-insert "/personal/" ["personal.org" ha/autoinsert-yas-expand])

Where ha/autoinsert-yas-expand basically calls yas-expand-snippet (search in this file for it). Now, we just need a helper function for throwing subtrees into that directory box:

(defun org-refile-to-personal ()
  "Move the current subtree to a file in the `personal' directory."
  (interactive)
  (org-refile-subtree-to-file org-default-personal-dir))

Summary

Finally, let’s clear the screen, and start my workflow:

(defun org-boxes-workflow ()
  "Load the default tasks file and start our hydra on the first task shown."
  (interactive)
  (let ((org-startup-folded nil))
    (find-file org-default-tasks-file)
    (delete-other-windows)
    (split-window-right-and-focus)
    (dired org-default-projects-dir)
    (pop-to-buffer (file-name-nondirectory org-default-tasks-file))
    (goto-char (point-min))
    (org-next-visible-heading 1)
    (hydra-org-refiler/body)))