Getting Boxes Done, the Code
Benjamin Franklin once wrote,
“For every minute spent in organizing, an hour is earned.”
Due to Emacs, I may have accrued nearly a lifetime by now.
As I mentioned in the first part of this essay, my shiny idea syndrome filled my task list with half-baked ideas, shopping lists, books to read, and solutions to all the world’s problems.
Assuming you like the idea of storing your project ideas in Org, the rest of essay describes the details and the code I use. Since the code for this project is tangled from this essay, ala literate programming, this page should stay up-to-date. ;-)
Workflow with an Example
As I mentioned before, I pretend ideas are on a virtual index card, and process them as I move them around organizational boxes. After creating a flowchart, I simplified it into more of a workflow:
What are the red letters, you ask? Those are the key bindings I use to move the virtual index cards from my inbox to other boxes (files or directories). You’ll see where I specify these and their implementation later.
Perhaps an example of using this workflow would be helpful.
I may notice that I’m running low on personal supplies, and quickly jot a note in Orgzly:
This puts a line in my Inbox:
* Buy toothpaste and eye drops
The next morning, as I’m reviewing the entries, I hit
t to refile (move) the
entry into my Task List. After I’ve been shopping, and later review the task
list, I hit
a to archive it, or
c to archive it the completed file, which for
me, is that day’s journal entry (however, you could specify a file like,
accomplishments-yay-me.org). Completed tasks in org are typically archived
and I discuss the details of this later on.
How about another example? The other day while noticing the large number of uncommitted text files (why yes, my org directories are under git control), I thought of an idea. I quickly kick off an Org capture and put another entry in my Inbox:
* Can I do a git commit automatically (at least, more regularly)?
When reviewing my Inbox the next day, I realize, while a good idea, I have no
clue how to accomplish it. Since it isn’t ready, I refile it to my
file (by typing
i). Returning to my incubating entries I hatch a plan
(which in this example involves a little web searching). With a plan, the wish
becomes a project, I update the entry:
* TODO Git commit on Sleep SCHEDULED: <2018-12-28 Fri> Certain directories under git-control often go weeks or months between commits, as "projects" containing text notes don't really have a *change* to commit. What about a checkpoint? Perhaps closing laptop or system sleeps, we commit all files. * On a Mac, checkout [[https://www.bernhard-baehr.de/][SleepWatcher]] * On Linux, checkout [[https://launchpad.net/ubuntu/+source/pm-utils][pm-utils]] Goal should be to have a script that both programs can run, e.g. #+BEGIN_SRC shell ~/bin/sleepwatcher -d -s ~/bin/on-sleep #+END_SRC
TODO label and scheduled date, as these make it show on my agenda.
As a real project, I refile it from
t, or if it’s a
larger, involved project, to its own file in
m P. Completion of
a project moves it again, but since this project contains notes I may have to
reference later, I shuffle the entire file to my
Why so many steps and so many files? In a large project management system, tasks are typically associated with states in a database where views present tasks in different states. I’m working with text files, so tasks in different states live in different files. Normally, this would be painful, but not with org-mode. Single key sequences shuffle the text to different files, allowing me to view state by simply viewing a file.
If this concept is clear, the remaining sections include my procedures (and Emacs functions) for processing ideas from inception to completion using the examples described above.
See all those destinations in the workflow illustration above? Let’s create constant variables, where the functions can use them later:
(defvar org-default-projects-dir "~/projects" "Primary GTD directory") (defvar org-default-technical-dir "~/technical" "Directory of shareable notes") (defvar org-default-personal-dir "~/personal" "Directory of un-shareable, personal notes") (defvar org-default-completed-dir "~/projects/trophies" "Directory of completed project files") (defvar org-default-inbox-file "~/projects/breathe.org" "New stuff collects in this file") (defvar org-default-tasks-file "~/projects/tasks.org" "Tasks, TODOs and little projects") (defvar org-default-incubate-file "~/projects/incubate.org" "Ideas simmering on back burner") (defvar org-default-completed-file nil "Ideas simmering on back burner") (defvar org-default-notes-file "~/personal/general-notes.org" "Non-actionable, personal notes") (defvar org-default-media-file "~/projects/media.org" "White papers and links to other things to check out")
Why yes, if you steal my code, you’ll want to change those values.
Oh, and since some of these are pretty cool and could have items that would
show up on an agenda, we may want to add them to
(add-to-list 'org-agenda-files org-default-inbox-file) (add-to-list 'org-agenda-files org-default-tasks-file)
Or, you can use code like this to add all files in the projects directory as potential agenda files:
(setq org-agenda-files (list org-default-projects-dir)) (setq org-agenda-file-regexp "^[a-z0-9-_]+.org")
However, let’s begin by keeping it simple:
(dolist (agenda-file (list org-default-inbox-file org-default-tasks-file org-default-incubate-file org-default-notes-file)) (add-to-list 'org-agenda-files (expand-file-name agenda-file)))
Everything begins as an entry in the Inbox (
org-default-inbox-file) that I
breathe.org (seemed like a good filename to avoid feeling overwhelmed),
so I need a flexible capturing entry to make it easy to create entries in
(add-to-list 'org-capture-templates `("t" "Task Entry" entry (file ,org-default-inbox-file) "* %?\n:PROPERTIES:\n:CREATED:%U\n:END:\n\n%i\n\nFrom: %a" :empty-lines 1))
org-capture (which most people bind to
C-c c, but Spacemacs has as
SPC C c)
might look like the following screenshot:
Note: With Orgzly (or other mobile org-specific application) configured to use this file as its default destination for all new ideas or todos, your mobile device will match your system.
Tidying by Refiling
As a temporary holding spot for all incoming tasks, notes and ideas, the goal is to keep the Inbox empty. Every day, I review the collected entries, and adjust and move them to appropriate locations.
I have created a little helper system for processing these ideas through the workflow shown in the flowchart above. Here is a screenshot of it in action where I am currently determining what to do with an entry captured earlier. The menu at the bottom of the screen matches my workflow shown earlier:
Note: The screenshot above may not match the code you’ll see below. Since this essay is really just a literate programming approach to generating the code I actually use, I’ll be changing the code, but may not re-render the images. Sorry.
While Emacs has many options for calling functions, I’ve chosen to use the
Hydra project to present a focused collection of keybindings to functions.
Originally I made a minor mode, but found Hydra’s hint screen and repetitive
key calls easier (once I start the hydra, I can simply type
i multiple times to
refile multiple entries to the incubation chamber).
Here is the definition for the Hydra, and you’ll notice that after the hint display, it is just a list of keybindings to functions:
(defhydra hydra-org-refiler (org-mode-map "C-c s" :hint nil) " ^Navigate^ ^Refile^ ^Move^ ^Update^ ^Go To^ ^Dired^ ^^^^^^^^^^--------------------------------------------------------------------------------------- _k_: ↑ previous _t_: tasks _m X_: projects _T_: todo task _g t_: tasks _g X_: projects _j_: ↓ next _i_/_I_: incubate _m P_: personal _S_: schedule _g i_: incubate _g P_: personal _c_: archive _p_: personal _m T_: technical _D_: deadline _g x_: inbox _g T_: technical _d_: delete _r_: refile _R_: rename _g n_: notes _g C_: completed " ("<up>" org-previous-visible-heading) ("<down>" org-next-visible-heading) ("k" org-previous-visible-heading) ("j" org-next-visible-heading) ("c" org-archive-subtree-as-completed) ("d" org-cut-subtree) ("t" org-refile-to-task) ("i" org-refile-to-incubate) ("I" org-refile-to-another-incubator) ("p" org-refile-to-personal-notes) ("r" org-refile) ("m X" org-refile-to-projects-dir) ("m P" org-refile-to-personal-dir) ("m T" org-refile-to-technical-dir) ("T" org-todo) ("S" org-schedule) ("D" org-deadline) ("R" org-rename-header) ("g t" (find-file-other-window org-default-tasks-file)) ("g i" (find-file-other-window org-default-incubate-file)) ("g x" (find-file-other-window org-default-inbox-file)) ("g c" (find-file-other-window org-default-completed-file)) ("g n" (find-file-other-window org-default-notes-file)) ("g X" (dired org-default-projects-dir)) ("g P" (dired org-default-personal-dir)) ("g T" (dired org-default-technical-dir)) ("g C" (dired org-default-completed-dir)) ("[\t]" (org-cycle)) ("s" (org-save-all-org-buffers) "save") ("<tab>" (org-cycle) "toggle") ("q" nil "quit"))
We just need to bind something like
C-c s to call
Or, under Spacemacs with a leader key:
(spacemacs/set-leader-keys-for-major-mode 'org-mode "r" 'hydra-org-refiler/body)
Or Doom’s leader key:
(map! :map org-mode-map :localleader :desc "boxes" "B" 'hydra-org-refiler/body)
Now I have a system for calling all these mover and shaker functions. Some
org-schedule, come with Org, while others, I’ll need to
Ideas in Incubation:
Things and ideas that need fleshing before tackling go into an incubation
file. Org has a nice feature for moving a subtree (an Org header entry and
its contents) to another file,
org-refile (see the Org Manual for details).
org-refile function presents a list of target files, which you
need to program using an odd
cons entry. I found specifying these easier to
learn by example. Along with the target file, you need to specify the heading
levels presented as a destination. Since I want top-level entries from Inbox to
go to top-level entries in the Incubation file, I specify a level of 0:
(setq org-refile-targets `((,org-default-incubate-file :level . 0)))
Not familiar with that back-tick/comma syntax? You can define a list of symbols with
'(a b c), where
ais not a variable, but just a Lisp symbol. But what if you wanted the
bto actually refer to the value of a variable? For this, you replace the normal tick
'(apostophe or single quote) with a back-tick,
`(or back quote), which acts the same, except that anything that follows a comma is a variable, and Emacs replaces it with its value. I guess I could have written that code as:(setq org-refile-targets (list (list org-default-incubate-file :level . 0)))
But now I am getting off topic… sorry for that interlude.
Anyway, to allow refiling entries to
incubate.org as a top-level
destination, we need to make two variable changes:
(setq org-refile-use-outline-path 'file org-refile-allow-creating-parent-nodes t org-outline-path-complete-in-steps nil)
Now, move to an org-mode section, and call
org-refile (with either
C-c C-w or
, s r if you are using Spacemacs or
SPC m r r in Doom … wait, I shouldn’t bother with all the keystrokes, as you, gentle reader, can look that up with
C-h f or
SPC h f if you are using Spacemacs or Doom… and here I did it again).
incubate.org shows this as a destination.
Since I last updated this essay, I realize that I incubate a lot of ideas, and I figured I might want to expand the incubating refiling to any file that starts with
(defun boxes--get-incubators () "Calls `completing-read' with a multiple choice of incubating files. That is, any files located in the `org-default-projects-dir' directory that begin with `incubate-'." (let ((choices (cons (cons "incubator" org-default-incubate-file) (--map (cons (replace-regexp-in-string ".*incubate-\\(.*\\).org" "\\1" it) it) (directory-files org-default-projects-dir t "incubate-.*.org"))))) (assoc (completing-read "Refile to Incubator:" choices) choices 'equal))) (defun org-refile-to-another-incubator (incubator) "Refile the current org subtree to an incubate file. A list of available `incubate-*.org' files are presented as a choice." (interactive (list #'boxes--get-incubators)) (org-refile nil nil (list (car incubator) (cdr incubator) nil 0)))
Well-developed tasks go into the
tasks.org file (which will show on my agenda):
(setq org-refile-targets `((,org-default-incubate-file :level . 0) (,org-default-tasks-file :level . 0)))
Books to read and music to check out go into appropriate sub-sections in my
media.org file, so I set
1 for this target destination:
(setq org-refile-targets `((,org-default-incubate-file :level . 0) (,org-default-media-file :level . 1) (,org-default-tasks-file :level . 0)))
Now, refiling should look something like:
Refile subtree to: incubate.org/ media.org/Articles media.org/Books media.org/Music media.org/Movies media.org/Series media.org/Games tasks.org/
However, every file in my
projects directory could be a potential destination,
and with the hydra making typical destinations easy, I could specify them all:
(setq org-refile-targets (append `((,(expand-file-name org-default-media-file) :level . 1) (,(expand-file-name org-default-notes-file) :level . 0)) (->> (directory-files org-default-projects-dir nil ".org") (-remove-item (file-name-base org-default-media-file)) (--remove (s-starts-with? "." it)) (--remove (s-ends-with? "_archive" it)) (--map (format "%s/%s" (expand-file-name org-default-projects-dir) it)) (--map `(,it :level . 0))))) (setq org-refile-target-table nil)
I am taking advantage of a threading macro from Magmar’s lovely dash library.
In case you are unfamiliar with the
->> macro, this code gets a list of all
the Org files, then removes the media file and any hidden files. It then takes
that list of files and creates a new list of fully-qualified file names and
0. Once you get the idea of the threading macro, the
code seems much easier to read.
If you haven’t installed dash,
subr-x library, and call the more verbose
thread-last macro, as it does the same thing.
I can’t call
org-refile programmatically with a file destination (as needed
in the hydra shown above), as it is only interactive. More problematic,
org-refile is a monolithic function, so I can’t call any helper functions it
might use. I currently see no other approach but to implement my own simpler
Copying regions is what Emacs does well…so, let’s define a region of a subtree:
(defun org-subtree-region () "Return a list of the start and end of a subtree." (save-excursion (list (progn (org-back-to-heading) (point)) (progn (org-end-of-subtree) (point)))))
Now we can use that function to kill the region of a subtree, open a file (the target destination), and insert the previous contents:
(defun org-refile-directly (file-dest) "Move the current subtree to the end of FILE-DEST. If SHOW-AFTER is non-nil, show the destination window, otherwise, this destination buffer is not shown." (interactive "fDestination: ") (defun dump-it (file contents) (find-file-other-window file-dest) (goto-char (point-max)) (insert "\n" contents)) (save-excursion (let* ((region (org-subtree-region)) (contents (buffer-substring (first region) (second region)))) (apply 'kill-region region) (if org-refile-directly-show-after (save-current-buffer (dump-it file-dest contents)) (save-window-excursion (dump-it file-dest contents)))))) (defvar org-refile-directly-show-after nil "When refiling directly (using the `org-refile-directly' function), show the destination buffer afterwards if this is set to `t', otherwise, just do everything in the background.")
After moving a subtree, do I want to see the resulting buffer in a window? If
so, I use
save-current-buffer (shown above), otherwise, this function calls
save-window-execursion. I haven’t decided which I like best, so I have a
customization variable that I can change.
Now, let’s create functions for the most-used refile destinations used by the
Hydra (notice that the Hydra can also call
org-refile directly in order to
get access to all targets):
(defun org-refile-to-incubate () "Refile (move) the current Org subtree to `org-default-incubate-fire'." (interactive) (org-refile-directly org-default-incubate-file)) (defun org-refile-to-task () "Refile (move) the current Org subtree to `org-default-tasks-file'." (interactive) (org-refile-directly org-default-tasks-file)) (defun org-refile-to-personal-notes () "Refile (move) the current Org subtree to `org-default-notes-file'." (interactive) (org-refile-directly org-default-notes-file)) (defun org-refile-to-completed () "Refile (move) the current Org subtree to `org-default-completed-file', unless it doesn't exist, in which case, refile to today's journal entry." (interactive) (if (and org-default-completed-file (file-exists-p org-default-completed-file)) (org-refile-directly org-default-completed-file) (org-refile-directly (get-journal-file-today))))
Scheduling and Planning
While reviewing the collected ideas in my Inbox, I often need to tidy them
before moving them around. Add a TODO label to each task with
T in my hydra,
as well as schedule a date with an
S (as a task without due date is just a
wish). Before I move the subtree, I may need to change the header’s text
(which I added to the hydra with an
(defun org-rename-header (label) "Rename the current section's header to LABEL, and moves the point to the end of the line." (interactive (list (read-string "Header: " (substring-no-properties (org-get-heading t t t t))))) (org-back-to-heading) (replace-string (org-get-heading t t t t) label))
Completed tasks are typically archived, using
org-archive-subtree. This is a
brilliant feature for keeping your notes relevant (as long as this feature
doesn’t surprise you by an accidental fat-fingering of the keyboard):
To archive to a separate file like this, set the variable,
However, in my world tasks marked as completed originate from multiple file
locations; I would rather move these finished tasks to my journal. I use
org-journal with a file for each day; for instance,
20180113 (without the
.org extension…see my journal helper functions if you are curious).
This function archives a subtree to today’s journal entry (marking the task completed in the process):
(defun org-archive-subtree-as-completed () "Archives the current subtree to today's current journal entry." (interactive) ;; According to the docs for `org-archive-subtree', the state should be ;; automatically marked as DONE, but I don't notice it, so let's force: (when (not (equal "DONE" (org-get-todo-state))) (org-todo "DONE")) (let* ((org-archive-file (or org-default-completed-file (todays-journal-entry))) (org-archive-location (format "%s::" org-archive-file))) (org-archive-subtree)))
The following function returns the filename of today’s journal entry (which I can use as a refile destination):
(defun todays-journal-entry () "Return the full pathname to the day's journal entry file. Granted, this assumes each journal's file entry to be formatted with year/month/day, as in `20190104' for January 4th. Note: `org-journal-dir' variable must be set to the directory where all good journal entries live, e.g. ~/journal." (let* ((daily-name (format-time-string "%Y%m%d")) (file-name (concat org-journal-dir daily-name))) (expand-file-name file-name)))
Refiling to Org Files
Archiving completed org entries (subtrees) is common, but sometimes a project entry grows and deserves its own file. While Org can refile or archive a subtree to a new file, these functions simply create a new file and copy the subtree. Personally, I really want a subtree to become a proper org file.
Writing that code became large and perhaps a bit of a white rabbit hole for most readers of this essay. Interested? Please, check out Part 3 of this essay.
We now have a Hydra that easily kicks off functions that help keep my task inbox manageable. Each morning, I would like to start an environment when my Inbox is loaded and ready for refiling. The following function does that by calling functions I would normally do manually:
(defun org-boxes-workflow () "Load the default tasks file and start our hydra on the first task shown." (interactive) (let ((org-startup-folded nil)) (find-file org-default-inbox-file) (delete-other-windows) (ignore-errors (ha/org-agenda)) (delete-other-windows) (split-window-right-and-focus) (pop-to-buffer (get-file-buffer org-default-inbox-file)) (goto-char (point-min)) (org-next-visible-heading 1) (hydra-org-refiler/body)))
This assumes that I have a special agenda display:
(defun ha/org-agenda () "Displays my favorite agenda perspective." (interactive) (org-agenda nil "a") (get-buffer "*Org Agenda*") (execute-kbd-macro (kbd "A t")))
I hope you can grab and modify this code to suit your own needs and whims. I may attempt to make the code more general and useful to others, but at the moment, I’d like to see if this code has any legs.
Let me know if you have any improvements you’d make.