www.howardism.org
Babblings of an aging geek in love with the Absurd, his family, and his own hubris.... oh, and Lisp.

Death to the Shell

I hate the Shell

piper-presentation-crab-shell.jpg

— Howard Abrams @howardabrams

Well, I kinda hate the Shell

As Emacsians, we have a love/hate relationship with shells.

  • shell, term, ansi-term, eshell, vterm, oh my…
  • Like dired, we have better options
piper-presentation-complicated.jpg

Agenda

I have two shell-related itches to scratch related to both shell uses:

  1. Interactive Data transformations through pipes are good and bad.
  2. Automative Personally, I want to replace 20+ years of shell scripts with Emacs Lisp.

Work in Progress

piper-presentation-wip2.png

This project is for tinkers only (at least, at this point).

Let’s Start with a Story

  • Hey Howard, can you restart the Openstack service for me?
  • Uh, you mean, services?

    for S in $(systemctl --all | grep openstack | sed 's/\.service.*//' | cut -c3-)
    do
        systemctl restart $S
    done
    

    That was easy to bang out, right?

    Converting data by chaining small executables is incredibly flexible.

Bad Parts of Pipes

The flow of data through pipes is inscrutable.

Similar to function calls: a | b | c(c (b (a)))

Good thing we can use an editor to do the real dirty work, eh?

$ systemctl --all > /tmp/all-services
$ emacsclient /tmp/all-services
$ systemctl restart $(cat /tmp/all-services)

Wait a minute…

Merging Shell Pipes and Emacs

In the shell, we use commands, editors, and scripts based on personal preference.

This flexibility gives a shell its power.

But Emacs has this same flexibility.

First Idea:

Transform data flows through pipes from command to command to Transform a buffer of data from function to function

Sending Data to Commands

Transforming data in an Emacs buffer is easy.

How do we use that data in the shell?

  • As standard in to a command
  • As a series of command line arguments (aka xargs)
  • Repeatedly run command with line as argument (aka for loop)
  • Copy data to clipboard
  • Just to visually inspect it (aka less)

Demonstration: Piper

  • Primarily a user interface
  • Thin wrapper around existing Emacs functions

Steps:

  • Cat /proc/acpi/wakeup
  • Pair it down to just the enabled device names
  • Write each device back to /proc/acpi/wakeup

Steps:

  • Get a list of all services: service --status-all
  • Filter to just the service names.
  • Get a description of each service.

Note: We can talk about a better name later. ☻

Scripts

First way we use the shell is interactively.

The second way is programatically … you know, scripting.

Now, Let’s talk about the second way.

Reddit Question

piper-presentation-a.png

And responded …

piper-presentation-b.png

Emacs Lisp as Go To Language?

  • Emacs is an ecosystem of Lisp functions
  • Emacs Lisp is really hackable
  • As a Lisp, you can port great ideas:
  • Good user interface:
    • Bind your program to a key chord/sequence
    • Completing read with Ivy or Helm
    • Hydra and Transient

Shell Scripts are both good and bad

Good Bad —————————————— ———————————————————

  • Can be readable • Can be unreadable
  • Executing programs • Hopefully no spaces in your filenames
  • Pretty good at data streams • Awful at data structures

Can we take the best of Lisp and Shell scripts?

As an Emacsian, calling a function is better than a script!

Maybe

What could we bring over from shells into Emacs Lisp?

  • Variables in strings:
  • Short, iterative, commands:

What could we bring over from Emacs Lisp?

  • Data structures
  • Better iterations
  • Better variable scoping
  • Functions, functions and more functions!

Desired Features

We will create a couple of macros to make Lisp look more scripty:

(piper-script
   @forms)

(defpiper-script name (params)
   :configuration
   @forms)

Shell Executables should just Work

(piper-script
 (shell "ls /tmp"))

And:

(piper-script
 (shell "ls" "/tmp"))

How about a short-cut?

(piper-script
 ($ "ls /tmp"))

Pipes and Data Transformations

Call a series of executable commands:

(piper-script
 ($ "ls" "-1 /tmp")
 (| "grep Temp")
 (| "tr [a-z] [A-Z]"))

Returns the string:

TEMP-19531244-A27D-4890-8D90-D6FDA91B6147
TEMP-F3C3C3F6-029D-442E-9441-6C676CEC3B62

The pipes between standard in and out, is just a temporary Emacs buffer.

Any other Emacs function can work too.

Wildcard Expansion

File expansion should work as expected:

(piper-script
 ($ "ls" "~/.emacs.d/*.el"))

Every string inside the piper-script automatically get converted with file-expand-wildcards.

(defun piper-expand (s)
  "Returns result of `file-expand-wildcards' if non-nil, otherwise, returns S."
  (or (file-expand-wildcards s) (list s)))

Embedded Strings

While format is fine, string-variable interpolation is more readable:

(let ((some-var "fooey"))
  (piper-script
   (let ((nudder-var "chop"))
     (echo "$HOME/projects/${nudder-var}/${some-var}"))))

Should return:

"/home/howard/projects/chop/fooey"

Notice we didn’t need to call get-env for the HOME value either.

Shell-Like Commands?

To make converting shell script to Emacs functions, should:

  • Write keep-lines as grep?
  • Write flush-lines as grep-v?
  • Use make-directory or mkdir?

I would like sudo to be a let wrapper around a modification to convert default-directory.

First Example

Let’s see process information about Google Chrome and Firefox…

My Original Script

Yeah, ps is flexible, but never flexible enough:

ps -u $USER -o ppid,rss,command | egrep 'Chrome|Firefox' | grep '^ *1' | grep -v Frameworks

Ya gotta love a series of regular expressions …

New Piper Script

Ooo, let’s use Emacs’ rx macro!

(let ((browsers      (rx (or "Chrome" "Firefox")))
      (session-leads (rx line-start (one-or-more blank) "1")))

  (piper-script
   ($ "ps" "-u $USER" "-o ppid,rss,command")
   (grep browsers)
   (grep session-leads)
   (grep-v "Frameworks"))))

Currently returns the string:

1 361356 /Applications/Firefox 2.app/Contents/MacOS/firefox
1 195164 /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Second Example

My laptop will turn on regularly after I close the lid.

Fix is simple, cat the “file” /proc/acpi/wakeup:

Device	S-state	  Status   Sysfs node
P0P2	  S3	*enabled  pci:0000:00:01.0
PEG1	  S3	*disabled
EC	  S4	*disabled  platform:PNP0C09:00
GMUX	  S3	*disabled  pnp:00:03
...
RP05	  S3	*disabled  pci:0000:00:1c.4
XHC1	  S3	*enabled  pci:0000:00:14.0
ADP1	  S4	*disabled  platform:ACPI0003:00
LID0	  S4	*enabled  platform:PNP0C0D:00

Write the device name back to the “file” to toggle it:

echo LID0 > /proc/acpi/wakeup

My Original Script

The following script disables any ’device events’ that can wake my laptop:

cat "/proc/acpi/wakeup" | grep "enabled" | sed "s/ .*//" | while read DEVICE
do
  echo $DEVICE > /proc/acpi/wakeup
done

New Piper Script

Re-written with piper-script:

(piper-script
 (sudo
  (cat "/proc/acpi/wakeup")
  (grep "enabled")
  (replace-regexp " .*" "")
  (dolist (device (read-all-lines))
    (write-into device "/proc/acpi/wakeup"))))

Show us the Magic

The magic is obviously a nifty macro.

(defmacro piper-script (&rest forms)
  "Runs the FORMS in a shell-like DSL."
  (cons 'with-temp-buffer
        (-tree-map #'piper--script-transform forms)))

Since all command works with a buffer, we create one.

Next, we potentially change every element in the tree…

Piper Script Transform

(defun piper--script-transform (element)
  "Helper for `piper-script' to convert forms and strings. "
  (cond
   ((stringp element) `(piper-script-get-files ,element))

   ((eq element '$) 'piper-script-shell)
   ((eq element '|) 'piper-script-shell)

   ((eq element 'echo) 'piper-script-echo)
   ((eq element 'ifsh) 'piper-script-shell-if)
   ((eq element 'touch) 'piper-script-touch)

   ((eq element 'ln-s) 'make-symbolic-link)
   ((eq element 'mkdir) 'make-directory)
   ((eq element 'cat) 'insert-file-contents)
   ((eq element 'write-into) 'piper-script-write-into)
   ((eq element 'to-clipboard) 'piper-script-to-clipboard)
   ((eq element 'read-all-lines) 'piper-script-read-all-lines)
   ;; ...
   (t element)))

Summary

  • Transforming data from standard in to standard out is better in an Emacs buffer
  • Sending data to an executable can be improved:
    • for loops
    • xargs, etc.
  • Any Lisp is better than Shell’s language:
    • Emacs Lisp is pretty hacky, and that’s a good thing
    • Calling Emacs functions is trivial and pretty flexible
    • Making Emacs Lisp as your go to scripting language is pretty fun

Discussion

Some potential sources for inspiration:

(defun emacs-piper-presentation-reset ()
  (interactive)
  (setq piper--command-history '())
  (setq history-delete-duplicates t)
  (add-to-history 'piper--command-history "cut -f1")
  (add-to-history 'piper--command-history "service --status-all"))

(use-package demo-it
  :load-path "~/Other/demo-it"
  :config
  (demo-it-create :advanced-mode :single-window
                  (demo-it-presentation (buffer-file-name) 3 :both)))