Benjamin Franklin once wrote, “For every minute spent in organizing, an hour is earned.” Due to Emacs, I think I have accrued nearly a lifetime by now.
As I mentioned in the first part of this essay, my shiny idea syndrome filled my task list with half-baked ideas, shopping lists, books to read, and solutions to all the world’s problems.
Assuming you like the idea of storing your project ideas in Org, the rest of essay describes the details and the code I use. Since the code for this project is tangled from this essay, ala literate programming, this page should stay up-to-date. ;-)
Workflow with an Example
As I mentioned before, I pretend ideas are on a virtual index card, and process them as I move them around my organizational boxes. After creating a flowchart, I simplified it into more of a workflow:
Perhaps an example of using this workflow would be helpful.
My captured entries begin life with a single line in the Inbox,
breathe.org, for example:
* Can I do a git commit automatically (at least, more regularly)?
Which I then refiled to my
incubate file for fleshing out. Returning to the
entry to hatch a plan (which often involves a little web searching).
With a plan, the wish becomes a project:
* TODO Git commit on Sleep SCHEDULED: <2018-12-28 Fri> Certain directories under git-control often go weeks or months between commits, as "projects" containing text notes don't really have a *change* to commit. What about a checkpoint? Perhaps closing laptop or system sleeps, we commit all files. * On a Mac, checkout [[https://www.bernhard-baehr.de/][SleepWatcher]] * On Linux, checkout [[https://launchpad.net/ubuntu/+source/pm-utils][pm-utils]] Goal should be to have a script that both programs can run, e.g. #+BEGIN_SRC shell ~/bin/sleepwatcher -d -s ~/bin/on-sleep #+END_SRC
TODO label and scheduled date, as these make it show up on my agenda.
As a real project, I move it from
incubate to either
tasks, or (if it is a
large, involved project), its own file in
projects. Completion of a project
moves it again, to either the
trophies directory, or if it has notes worth
remember, I shuffle it to the
technical directory for safe-keeping.
Why so many steps and so many files? In a large project management system, tasks are typically associated with states in a database where views present tasks in different states. I’m working with text files, so tasks in different states live in different files. Normally, this would be painful, but not with org-mode.
If this concept is clear, the remaining sections include my procedures (and Emacs functions) for processing my ideas from inception to completion.
See all those destinations in the workflow? Let’s create constant variables, where the functions can use them:
(defvar org-default-projects-dir "~/projects" "Primary GTD directory") (defvar org-default-technical-dir "~/technical" "Directory of shareable notes") (defvar org-default-personal-dir "~/personal" "Directory of un-shareable, personal notes") (defvar org-default-inbox-file "~/projects/breathe.org" "New stuff collects in this file") (defvar org-default-tasks-file "~/projects/tasks.org" "Tasks, TODOs and little projects") (defvar org-default-incubate-file "~/projects/incubate.org" "Ideas simmering on back burner") (defvar org-default-notes-file "~/personal/general-notes.org" "Non-actionable, personal notes") (defvar org-default-completed-dir "~/projects/trophies" "Completed projects") (defvar org-default-media-file "~/projects/media.org" "White papers and links to other things to check out")
Everything begins as an entry in the INBOX (
org-default-inbox-file) that I
breathe.org (seemed like a good filename to avoid feeling overwhelmed),
so I need a flexible capturing entry:
(add-to-list 'org-capture-templates `("t" "Task Entry" entry (file ,org-default-inbox-file) "* %?\n:PROPERTIES:\n:CREATED:%U\n:END:\n\n%i\n\nFrom: %a" :empty-lines 1))
Which might look like the following screenshot:
Configure Orgzly with this file as the default destination for all new ideas or todos. Then your mobile device will match your system.
Tidying by Refiling
As a temporary holding spot for all incoming tasks, notes and ideas, the goal is to keep this file empty. Every day, I review the collected entries, and adjust and move them to appropriate locations.
I have created a little helper system for processing these ideas through the workflow shown in the flowchart above. Here is a screenshot of it in action:
Note: The screenshot above won’t match the code you’ll see below. Since this essay is really just a literate programming approach to generating the code I actually use, I’ll be changing the code, but I’m not going to re-render the images. Sorry.
I need a collection of functions to easily archive or refile each subtree entry. A hydra can easily call these functions:
(defhydra hydra-org-refiler (org-mode-map "C-c s" :hint nil) " ^Navigate^ ^Refile^ ^Move^ ^Update^ ^Go To^ ^Dired^ ^^^^^^^^^^------------------------------------------------------------------------------------- _k_: ↑ previous _r i_: incubate _r x_: projects _T_: todo task _g b_: Inbox _p x_: Projects _j_: ↓ next _r j_: journal _r p_: personal _S_: schedule _g t_: Tasks _p p_: Personal ^ ^ heading _r n_: task _r t_: technical _D_: deadline _g i_: Incubate _p t_: Technical _d_: delete _r r_: refile other _R_: rename _g n_: Notes " ("<up>" org-previous-visible-heading) ("<down>" org-next-visible-heading) ("k" org-previous-visible-heading) ("j" org-next-visible-heading) ("g b" (find-file org-default-inbox-file)) ("g t" (find-file org-default-tasks-file)) ("g i" (find-file org-default-incubate-file)) ("g n" (find-file org-default-notes-file)) ("r i" org-refile-to-incubate) ("r j" org-refile-to-journal) ("r n" org-refile-to-task) ("r r" org-refile) ("r x" org-refile-to-projects) ("r p" org-refile-to-personal) ("r t" org-refile-to-technical) ("d" org-kill-subtree) ("S" org-schedule) ("D" org-deadline) ("T" org-todo) ("R" org-rename-header) ("p x" (dired org-default-projects-dir)) ("p p" (dired org-default-personal-dir)) ("p t" (dired org-default-technical-dir)) ("q" nil "quit"))
We just need to bind something like
C-c s to call
Or, under Spacemacs with a leader key:
(spacemacs/set-leader-keys-for-major-mode 'org-mode "r" 'hydra-org-refiler/body)
Note: Use Orgmode’s refiling feature for shuffling a header and its text
(called a subtree) from one file to another. While I begin discussing
getting stuff out of my initial collection inbox,
breathe.org, the following
section shows all the code for moving subtrees around.
This file in the
projects directory holds things and ideas that need fleshing…
basically, good ideas and potential projects. This should be a refile
(setq org-refile-targets '((org-default-incubate-file :level . 0)))
To allow entries (org-mode sections) called with
org-refile to have
as a top-level destination, however, we need to make two changes:
(setq org-refile-use-outline-path 'file org-outline-path-complete-in-steps nil)
To move an org-mode section, move the point (cursor) into it, and call
org-refile (with either
C-c C-w or
, s r if you are using Spacemacs), and notice
incubate.org shows as a destination.
Add any other destinations you might find helpful. Well thought tasks go into
tasks.org (which will show on my agenda):
(setq org-refile-targets '((org-default-incubate-file :level . 0) (org-default-tasks-file :level . 0)))
Books to read and music to check out go into appropriate sections in my
media.org file, so I set
1 for that destination:
(setq org-refile-targets '((org-default-incubate-file :level . 0) (org-default-media-file :maxlevel . 1) (org-default-tasks-file :level . 0)))
Completed Tasks go to the Journal
Completed tasks are usually archived, using
org-archive-subtree, however, I
would rather move completed tasks and random notes to my journal. This
function returns the filename of today’s journal entry to be used as a refile
(defun todays-journal-entry () "Return the full pathname to the day's journal entry file. Granted, this assumes each journal's file entry to be formatted with year/month/day, as in `20190104' for January 4th. Note: `org-journal-dir' variable must be set to the directory where all good journal entries live, e.g. ~/journal." (let* ((daily-name (format-time-string "%Y%m%d")) (file-name (concat org-journal-dir daily-name)) (journal-file (expand-file-name file-name)))) (list journal-file))
No, I’m not sure why functions used as targets for
org-refile need to return a
list. Anyway, here is the final list of all variable values:
(setq org-refile-use-outline-path 'file org-indent-indentation-per-level nil org-outline-path-complete-in-steps nil org-refile-targets (append '((org-default-media-file :maxlevel . 1) ;; Note: Any items listed here need to be removed below ;; Completed tasks go into today's journal entry: (todays-journal-entry :level . 0)) (->> (directory-files org-default-projects-dir nil ".org") (-remove-item (file-name-base org-default-media-file)) (--remove (s-starts-with? "." it)) (--map (format "%s/%s" org-default-projects-dir it)) (--map `(,it :level . 0)))))
Wait, what happened there, Howard?
After adding the
media with a level of 1, and the today’s journal, I then
added every file in the project directory by taking advantage of Magmar’s
lovely dash library.
org-refile shows me a goodly list of destinations for entries
Refiling destinations refer to existing files, but the hydra shown above
needs to be able to refile to directories as well. Also, I can’t call
org-refile directly with a destination; it is only interactive. More
org-refile is a monolithic function.
I currently see no other approach, but to implement my own simpler refiler. Copying regions is what Emacs does well…so, let’s define a region of a subtree:
(defun org-subtree-region () "Return a list of the start and end of a subtree." (save-excursion (list (progn (org-back-to-heading) (point)) (progn (org-end-of-subtree) (point)))))
Now kill that region, open a file destination, and insert it:
(defun org-refile-directly (file-dest &optional show-after) "Move the current subtree to the end of FILE-DEST. If SHOW-AFTER is non-nil, the destination window will be shown, otherwise, this destination buffer is not shown." (interactive "fDestination: ") (defun dump-it (file contents) (find-file-other-window file-dest) (goto-char (point-max)) (insert "\n" contents)) (save-excursion (let* ((region (org-subtree-region)) (contents (buffer-substring (first region) (second region)))) (apply 'kill-region region) (if show-after (save-current-buffer (dump-it file-dest contents)) (save-window-excursion (dump-it file-dest contents))))))
After moving a subtree, do I want to see the resulting buffer in a window?
If so, then I use
save-current-buffer, otherwise, call
Now, I can create functions for the most used refile destinations:
(defun org-refile-to-incubate () (interactive) (org-refile-directly org-default-incubate-file)) (defun org-refile-to-task () (interactive) (org-refile-directly org-default-tasks-file)) (defun org-refile-to-journal () (interactive) (org-refile-directly (get-journal-file-today)))
Scheduling and Planning
Add a TODO label to each task (with a
T in my hydra) as well as schedule a
date (with an
S), as a task without due date is just a wish. Before I move the
subtree, I may need other touchup, like easily changing the header (which I
added to the hydra with a
(defun org-rename-header (label) "Rename the current section's header to LABEL, and moves the point to the end of the line." (interactive (list (read-string "Header: " (substring-no-properties (org-get-heading t t t t))))) (org-back-to-heading) (replace-string (org-get-heading t t t t) label))
Not everything I store needs to be remember or acted. Some, like my shopping list, I delete after completing. Not sure why we don’t have a simple function to delete an org subtree section, but seems easy enough:
(defun org-kill-subtree () "Deletes an org section (subtree) including the header and all content it 'contains'. Move the content to the kill-ring." (interactive) (let ((region (org-subtree-region))) (kill-region (first region) (second region)) (org-next-visible-heading 1)))
Tidying to Files
Shuffling org entries (subtrees) from one file to another is common, but sometimes an entry grows and deserves its own file. Org doesn’t have this use case, so let’s make it.
First, I will need a function to collect the subtree’s goodies like the contents of the header and its body:
(defun org-subtree-metadata () "Return a list of key aspects of an org-subtree. Includes the following: header text, body contents, list of tags, region list of the start and end of the subtree." (save-excursion (let* ((header (substring-no-properties (org-get-heading t t t t))) (tags (progn (org-back-to-heading) (org-get-tags))) (del-tags (org-set-tags-to "")) (head-start (point)) (body-start (progn (forward-line) (point))) (body-end (progn (org-end-of-subtree nil t))) (body (buffer-substring-no-properties body-start body-end))) (list header body tags (list head-start body-end)))))
I need to choose a good filename to place the contents. Sure, I would like to
take the section’s header, however, what should I do with something like,
Let's go Shopping! … in this case, I’ll convert all the non-alphanumeric
characters to dashes and lowercase everything:
(defun org-filename-from-title (title) "Creates a useful filename based on a header string, TITLE. For instance, given the string: What's all this then? This function will return: whats-all-this-then" (let* ((no-letters (rx (one-or-more (not alphanumeric)))) (init-try (->> title downcase (replace-regexp-in-string "'" "") (replace-regexp-in-string no-letters "-")))) (string-trim init-try "-+" "-+")))
org-refile-subtree-to-file is the primary entry into this
concept. It takes a directory destination and moves the section to a new file
(defun org-refile-subtree-to-file (dir) "Archive the org-mode subtree and create an entry in the directory folder specified by DIR. It attempts to move as many of the subtree's properties and other features to the new file." (interactive "DDestination: ") (destructuring-bind (header body tags region) (org-subtree-metadata) (let* ((filename (org-filename-from-title header)) (filepath (format "%s/%s.org" dir filename))) (apply delete-region region) (org-archive-subtree-to-file filepath header body tags))))
The heavy lifting of the previous function is actually done by this function, which given all the information it needs, deletes the subtree, creates a new file, and updates it. Note: By updating the file, we can take advantage of directory of file-specific auto-insert.
(defun org-archive-subtree-to-file (filepath header body tags) "Use `org-archive-subtree' to move the current subtree to the file given by FILEPATH. This requires pre-knowledge of subtree, including the HEADER, BODY and any associated TAGS." (find-file-other-window filepath) (goto-char (point-min)) ;; Insert if not found: (replace-regexp "^#\\+TITLE:.*" (concat "#+TITLE: " header)) (re-search-forward "^\s*$") (when tags ;; TODO Search for TAGS: line and replace instead of inserting (insert (concat "#+TAGS: " (s-join " " tags) "\n\n"))) (replace-regexp "^\\* 0$" "") (org-archive-subtree-to-file-move-props)) (defun org-archive-subtree-to-file-move-props () "The `org-archive-subtree' function copies a subtree to the new file, leaving the subtree's properties under the heading, and these really should be moved to the top-level of the file. It is a bit complicated and ugly, but it is an ugly affair." (save-excursion (goto-char (point-min)) (re-search-forward "^\s*$" nil t) (setq property-location (point)) ;; Copy the section's properties to the top of the file: (org-next-visible-heading 1) (cl-loop for elem in (org-entry-properties) do (destructuring-bind (key . val) elem (when (not (--any? (equal key it) '("ITEM" "CATEGORY" "PRIORITY" "BLOCKED" "FILE"))) (goto-char property-location) (insert "#+PROPERTY: " key " " val "\n")))) ;; Delete the subtree header and its properties: (let ((start (progn (goto-char (point-min)) (re-search-forward "^$") (point))) (end (re-search-forward ":END:"))) (delete-region start end))))
auto-insert to pre-populate a file is great, so I need a way to make
sure certain lines are set correctly, and if not, insert them:
(defun org-set-file-property (property value &optional current-point) "Where PROPERTY is a top-level, file-wide property, like `TITLE' or `TAGS', this function makes sure that the property contains the contents of VALUE, and if the file doesn't have the property, it is inserted at either the CURRENT-POINT if non-nil, or at the next available blank line." (save-excursion (let* ((upprop (upcase property)) (rexpr (format "#\\+%s: .*" upprop)) (contents (format "#+%s: %s" upprop value)) (start (point)) (end (progn (re-search-forward "^\s*$") (point)))) (if (re-search-forward rexpr end) (replace-match contents) (goto-char end) (insert contents) (insert "\n")) (message rexpr) ))) (org-set-file-property "foo" "bling") ;; Let's try this ... point point on this line: ;; Then we should be able to replace: ;; foo: bar #+FOO: bar ;; However, we should not find badaboom at all. #+FOO: bling
I think I spent a bit too much time worry about text that can be easily manipulated afterwards. Well, now I can finish my workflow.
Good, big, chewy projects may start life as a single idea in
when they outgrow that file, they need to be moved to their own org file in
(defun org-refile-to-projects () "Move the current subtree to a file in the `projects' directory." (interactive) (org-refile-subtree-to-file org-default-projects-dir))
Completed projects need to be moved out of the
projects directory. This could be:
technical: where complicated notes can help with ongoing maintenance
personal: need to remember personal information about the project
projects/trophies: seems like an apt name for a
Sounds like a job for
technical folder contains any notes on non-work, non-personal
information. The idea with this box is that I can share it publicly.
(defun org-refile-to-technical () "Move the current subtree to a file in the `technical' directory." (interactive) (org-refile-subtree-to-file org-default-technical-dir))
Any thing to be remembered or referenced goes into a file in the
folder. Each of these files end with a
.txt extension so that Dropbox can
display it. However, it is still an org file, so my Yasnippet template for
it looks like:
--org-- #+TITLE: $1 #+AUTHOR: Howard Abrams #+EMAIL: firstname.lastname@example.org #+TAGS: personal $2 $0
We need an auto insert for anything in that directory to expand that snippet.
(define-auto-insert "/personal/" ["personal.org" ha/autoinsert-yas-expand])
ha/autoinsert-yas-expand basically calls
yas-expand-snippet (search in this file for it).
Now, we just need a helper function for throwing subtrees into that
(defun org-refile-to-personal () "Move the current subtree to a file in the `personal' directory." (interactive) (org-refile-subtree-to-file org-default-personal-dir))
Finally, let’s clear the screen, and start my workflow:
(defun org-boxes-workflow () "Load the default tasks file and start our hydra on the first task shown." (interactive) (let ((org-startup-folded nil)) (find-file org-default-tasks-file) (delete-other-windows) (split-window-right-and-focus) (dired org-default-projects-dir) (pop-to-buffer (file-name-nondirectory org-default-tasks-file)) (goto-char (point-min)) (org-next-visible-heading 1) (hydra-org-refiler/body)))