Presenting the Eshell

The following is a transcript of the slides and demonstration of EShell I gave to both the PDX and London Emacs User groups. Hopefully this page will be easier to copy/paste…


John Wiegley created EShell in 1998:

…as a way to provide a UNIX-like environment on a Windows NT machine.

Part of Emacs since v21.


  • Started with ksh
  • Used a lot of shells…
  • Tried eshell soon after its birth
  • Shelved it since it wasn’t shell-enough
  • Rediscovered years later
  • Finally got it

Contents: What’s all this then?

  • What EShell really is
  • How to use it effectively
  • Hacking

Shell… The Good

  • Can be immensely powerful… at times
  • Pipes and redirection are a staple
  • Utilizing small, focused text-oriented executables
  • Complex command re-invocation
  • History okay … nothing like Emacs

Shell… The Bad

  • Commands? Like key sequences, only longer
  • Needing completion to run commands?
  • Loops? Not terrible
  • Copy and pasting … with a mouse!? (at least use M-x shell)

Shell… The WTF?

  • Best part: extensibility!
  • But what an awful language:

    if [ $(echo "$IN" | cut -c 1-3 ) == 'abc' ]; then
       # ...
  • May be Turing complete, but so what.

    But, but, but… we know the shell!

Seen Rich Hickey’s Simple Made Easy talk? The shell is easy because it is so close.


Python REPL with shell-like features.

  • Understands a current directory
  • Has some shell-like commands, cat
  • Doesn’t easily execute programs: system
  • Executes Python scripts: run

Slide Notes

Demonstrated the iPython approach by entering the following into an ipython REPL:

Python 2.7.12 (default, Nov 19 2016, 06:48:10)
Type "copyright", "credits" or "license" for more information.

IPython 2.4.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 2 ** 60
Out[1]: 1152921504606846976

In [2]: cd /tmp/testing

In [3]: ls  pi.rb  README  src/  tests/

In [4]: cat
import math


In [5]: chmod a+x
  File "<ipython-input-5-b9e30b9b8e31>", line 1
    chmod a+x
SyntaxError: invalid syntax

In [6]: run pi

In [7]: system ruby pi.rb
Out[7]: ['-0.9999999999964793', '-3.141592653589793']

In [8]: cat README
Ciao, all you cool cats.
Oh, and hey to all my dawgs.

In [9]: def cat(arg=None):
   ...:     return 'Meow!'

In [10]: cat
Out[10]: <function>

In [11]: cat()
Out[11]: 'Meow!'

In [12]: cat README
  File "<ipython-input-12-01ccd53eccb4>", line 1
    cat README
SyntaxError: invalid syntax

In answer to your question, I haven’t looked to see why we have an array when calling the system function.

If you want to see what shell-like functions iPython has, type % and hit the Tab key, which shows something like:

In [13]: %
Display all 122 possibilities? (y or n)
%%!                      %doctest_mode            %pfile
%%HTML                   %ed                      %pinfo
%%SVG                    %edit                    %pinfo2
%%bash                   %env                     %popd
%%capture                %gui                     %pprint
%%debug                  %hist                    %precision
%%file                   %history                 %profile
%%html                   %install_default_config  %prun
%%javascript             %install_ext             %psearch
%%latex                  %install_profiles        %psource
%%perl                   %killbgscripts           %pushd
%%prun                   %ldir                    %pwd
%%pypy                   %less                    %pycat
%%python                 %lf                      %pylab
%%python2                %lk                      %quickref
%%python3                %ll                      %recall
%%ruby                   %load                    %rehashx
%%script                 %load_ext                %reload_ext
%%sh                     %loadpy                  %rep
%%svg                    %logoff                  %rerun
%%sx                     %logon                   %reset
%%system                 %logstart                %reset_selective
%%time                   %logstate                %rm
%%timeit                 %logstop                 %rmdir
%%writefile              %ls                      %run
%alias                   %lsmagic                 %save
%alias_magic             %lx                      %sc
%autocall                %macro                   %store
%autoindent              %magic                   %sx
%automagic               %man                     %system
%bookmark                %matplotlib              %tb
%cat                     %mkdir                   %time
%cd                      %more                    %timeit
%clear                   %mv                      %unalias
%colors                  %notebook                %unload_ext
%config                  %page                    %who
%cp                      %paste                   %who_ls
%cpaste                  %pastebin                %whos
%debug                   %pdb                     %xdel
%dhist                   %pdef                    %xmode
%dirs                    %pdoc

EShell as a Shell

  • Most “interactive language” interfaces choose:
    • Language-specific REPL, or
    • Shell-focused program worker
  • As a shell:
    • Concept of a current directory
    • popd, pushd, and dirs
    • Globbing Expressions
    • Quotes often optional
      • Do you care about spaces?
      • Double and single quotes are interchangeable
    • Aliases: alias ll 'ls -l'
  • Emacs shell interaction:
    • M-n / M-p scroll through history
    • M-r select from history
    • C-c C-p move to previous prompts
    • C-c C-l list history in buffer
  • Tempted to think eshell is like shell

Slide Notes

At this point, we start an eshell process, and demonstrate some of the shell-like features that we’d expect for something that ends with -shell:

$ cd /tmp/testing

$ pwd

$ ls
README  pi.rb  src  tests

$ cat README
Ciao, all you cool cats.
Oh, and hey to all my dawgs.

$ ruby pi.rb

$ python *.py

$ echo "Hello"

$ echo 'Hello'

$ echo Hello

$ alias ee 'find-file-other-window $1'

$ ee pi.rb
#<buffer pi.rb>

This last example shows that ee opens a window in another window. Note, however, that the alias is actually calling an Emacs function, not another executable (although it could).

EShell as a REPL

  • Lisp expressions work within parens
  • Unlike shell, EShell:
    • Commands can be executables or Emacs functions
    • Distinguishes strings, numbers, and lists
  • EShell is marriage of two syntax parsers:
    • Shell Expressions
    • Lisp Expressions
  • A single line can mix the two!

Slide Notes

To demonstrate how eshell is a REPL, let’s type a simple Lisp expression:

$ (length "hi")

$ length "hi"      # ← Works since shell parser calls Lisp

$ length hi        # ← Works since shell reads as string

$ (+ 1 3)

$ + 1 3            # ← Works since shell reads as number

$ * 3 (+ 1 2)      # ← Shows both shell and lisp parsers

$ ls
README  pi.rb  src  tests

$ length *        # Globs return a list

$ length *.py

$ touch 'and'

$ ls *.py

$ echo *.py        # ← More clear that globs are lists
("and" "")

Eshell’s Parsers

  • Lisp parser:
    • ( ... )
    • $( ... ) … useful for string evaluation
  • Shell parser:
    • no parens … in other words, the default
    • { ... }
    • ${ ... } … useful for string evaluation
  • In shell parser, reference variables with $

Slide Notes

To drive home the differences between shell and lisp parsers, let’s enter the following in eshell buffer:

$ setq ANSWER 42   # ← Normal Emacs variable

$ numberp $ANSWER  # ← Use $ to get value.

$ setq UNANSWER "41"

$ stringp $UNANSWER

$ mod ANSWER 5    # ← Forgot the $ with shell parser
Wrong type argument: number-or-marker-p, "ANSWER"

$ mod $ANSWER 5   # ← Math without expr

$ (mod ANSWER 5)  # ← Lisp doesn't need $ for vars

$ (mod $ANSWER 5)
Symbol's value as variable is void: $ANSWER


$ echo $UNANSWER:(mod ANSWER 5)   # ← Let's talk about predicate filters later in the show
Malformed modification time modifier `m'

$ echo $UNANSWER:$(mod ANSWER 5)

$ echo $ANSWER:${mod $ANSWER 5}

Shell-like Loops

  • Syntactic sugar around loop.
  • Code following in is a generate list
  • Use trailing { ... } for side-effects

Slide Notes

See this page for more details.

Show off the for concept, by entering the following in an eshell buffer. Note that the do syntax for some shells doesn’t work. Loops look more like csh’s:

$ for F in *; do echo "I like $F"; done
Symbol's value as variable is void: do
done: command not found

$ for F in * { echo "I like $F" }
I like and
I like
I like pi.rb
I like src/
I like tests/

# A list can be generated in any way, like with Lisp:
$ for N in (number-sequence 1 5) { % $ANSWER $N }

# Generate the list with eshell parser syntax:
$ for N in {number-sequence 1 5} { % $ANSWER $N }

# Note: Can't use Lisp as the loop's action:
$ for N in {number-sequence 1 5} (% ANSWER N)

# Unless you embed the Lisp in shell parser:
$ for N in {number-sequence 1 5} {(% ANSWER N)}

Function or Executable?

What about the executable find vs. Emacs’ find function?

Precedence Order:

  1. Eshell aliases
  2. Emacs functions that being with eshell/ prefix
  3. Normal Emacs functions (don’t need to be interactive)
  4. Shell executables

Of course, this is customizable:

  • eshell-prefer-lisp-functions prefer Lisp functions to external commands
  • eshell-prefer-lisp-variables prefer Lisp variables to environmentals

Slide Notes

To demonstrate the precedence order for eshell commands, I created a script called foobar that simply contains:


echo "Called: executable"

Without anything else, this will be called:

$ which foobar

$ foobar
Called: executable

We now create a regular Emacs function in Lisp (notice that it isn’t interactive):

(defun foobar ()
  "Called: function")

It now takes precedence over the executable:

$ which foobar
foobar is a Lisp function

$ foobar
Called: function

Create another Lisp function, this has the eshell/ prefix. Again, no need to make interactive:

(defun eshell/foobar ()
  "Called: eshell function")

And this new function over-shadows the others:

$ which foobar
eshell/foobar is a Lisp function

$ foobar
Called: eshell function

Finally, we define an alias, and demonstrate that it over-shadows all the others:

$ alias foobar 'echo "Called: alias"'

$ which foobar
foobar is an alias, defined as "echo "Called: alias""

$ foobar
Called: alias

Globbin’ Filters

  • The * glob-thing has filters
  • Great if you can remember the syntax:
    • . for files
    • / for directories
    • r if readable
    • w if writable
    • L filtering based on file size
    • m filtering on modification time
  • The filters can be stacked, e.g. .L
  • Can’t remember? C-c M-q Or: eshell-display-predicate-help

Slide Notes

Using a directory for this purpose, we can demonstrate EShell’s predicate filter feature. First, list all files:

$ ls *(.)
README  and  pi.rb

$ ls *(^/)   # ← Inverse of directories are often files
README  and  pi.rb

Demonstrate combining modifiers by listing all files with more than 50 bytes to them:

$ ls *(.L+50)

After creating three files (using the touch executable), we can list all empty files (that is, those that have less than 1 byte):

$ ls *(L-1)

Or those modified less than 40 seconds ago:

$ ls *(.ms-40)
README  and  pi.rb  src  tests

Modified after we modified

$ ls *(.m-'')

And before we modified

$ ls *(.m+'')
README  and  pi.rb

I love this. I can get a list of my journal entries larger than 5000 bytes, and open dired showing only those files:

$ dired ~/journal/2017*(L+5000)


Syntactic sugar to convert strings and lists.

Can’t remember? C-c M-m Or: eshell-display-modifier-help

Eshell filters and modifiers remind me of regular expressions

Don’t know the eshell-way? Just drizzle Lisp.

Slide Notes

Convert a string with a modifier:

$ echo "hello"(:U)

Convert all files, as strings, in a list:

$ echo *(:U)

Modifiers can be combined with filters:

$ echo *(.L-1:U)

However, we often split these and use the for loop:

$ for F in *(.L-1) { mv $F $F(:U) }

$ ls
AND GO.PY  GOO.PY  GRIP.PY  README  SWAM.PY  pi.rb  src  tests

Now, all empty files are in upper case.

You’d think you could reverse a list with:

$ echo ("hello" "cruel" "world")(:R)
No matches found: ("hello" "cruel" "world")

Since the shell parser doesn’t like that syntax, perhaps you could set the list to a variable and work with that?

$ setq BADDABING (list "hello" "cruel" "world")
("hello" "cruel" "world")

$ echo $BADDABING(:R)
("world" "cruel" "hello")

If you find this stuff too odd and confusing, you can always fall back to Lisp:

$ reverse (list "hello" "cruel" "world")
("world" "cruel" "hello")

EShell Hack Points

While offering similar shell experience, Eshell is really hackable!

Here are some ideas…

Write your Own Functions

  • Functions for Eshell: eshell/
  • They do not need to be interactive
  • Functions should assume &rest for arguments:

    (defun eshell/do-work (&rest args)
      "Do some work in an optional directory."
      (let ((some-dir (if args
                           (pop args)
        (message "Work in %s" some-dir)))

Slide Notes

Using &rest allows your functions to behave more like shell functions:

$ do-work
Work in /tmp/testing/

$ do-work /home/howard/bin
Work in /home/howard/bin

Remote Connections

To have eshell work on a remote server:

(let ((default-directory "/"))

My personal project:

  • Connect to my hypervisor controller
  • Download and store a list of virtual machines
  • Use ido-completing-read to select a host / ip
  • Generate a Tramp URL for default-directory

Slide Notes

Here is a simplified example that might be a helpful start:

(defvar eshell-fav-hosts (make-hash-table :test 'equal)
  "Table of host aliases for IPs or other actual references.")

(puthash "web-server" "" eshell-fav-hosts)
(puthash "slc-jumpbox" "" eshell-fav-hosts)
;; ...

(defun eshell-favorite (hostname &optional root dir)
  "Start an shell experience on HOSTNAME, that can be an alias to
a virtual machine from my 'cloud' server. With prefix command,
opens the shell as the root user account."
    (ido-completing-read "Hostname: "
                         (hash-table-keys (eshell-fav-hosts)))))

  (when (equal current-prefix-arg '(4))
    (setq root t))
  (when (not dir)
    (setq dir ""))

  (let* ((ipaddr (gethash hostname eshell-fav-hosts hostname))
         (trampy (if (not root)
                     (format "/ssh:%s:%s"       ipaddr dir)
                   (format "/ssh:%s|sudo:%s:%s" ipaddr ipaddr dir)))
         (default-directory trampy))

Extending Predicates

The User predicate (U) could have been written:

(defun file-owned-current-uid-p (file)
  (when (file-exists-p file)
      (= (nth 2 (file-attributes file))

Then add it:

(add-hook 'eshell-pred-load-hook (lambda ()
  (add-to-list 'eshell-predicate-alist
           '(?U . 'file-owned-current-uid-p))))

My engineering notebook is a directory of files.

Most of my files have #+tags entries.

I can filter based on these tag entries.

I have to parse text following predicate key.

Slide Notes

Assuming the script is owned by the root user, we can show which ones I own:

$ ls *(.U)

And which ones I don’t:

$ ls *(^U)

My engineering notes contains quite a few files:

$ length ~/technical/*(.)

For instance, one file in my engineering notebook starts with:

TITLE:  Perfect Square
AUTHOR: Howard Abrams
DATE:   2013 Jun 06
tags:   programming clojure

Haitao posed a question: How do you write a function that determines
if a number is a perfect square.  You know, 9 and 25 are both perfect
squares because their square roots are natural integers.

While I have a brute force approach with an imperative loop in my
head, I'm curious if I could do it with Lisp...

Notice the line that starts with #+tags: … I can get the files that contain a word on this line, with my new T predicate:

$ length ~/technical/*(.T'clojure')

Replacing Pipes

Pipes for shell are flexible, but…

  • Shell’s text processing is limited
  • Need arsenal of tiny, cryptic programs
  • Re-run many times since debug pipe steps

Emacs is pretty good at text processing

  • keep-lines / flush-lines instead of grep
  • replace-string, et. al instead of sed

In EShell, redirect output to Emacs buffer:

$ some-command > #<buffer buf-name>

After editing the buffer, use it:

$ bargs #<buf-name> mv % /tmp/testing

Reference buffers as #<buf-name> with:

(setq eshell-buffer-shorthand t)

Or use keybinding, C-c M-b

Slide Notes

To demonstrate this feature, I first put a string in a new buffer, fling:

$ echo hello > #<buffer fling>

And displayed the buffer contents just to be sure.

Next, let’s overwrite the contents of that buffer:

$ ls -1 > #<buffer fling>

Now I call keep-lines to choose only the python files, and flush-lines to remove all files that contain go.

To show that I can now get the remaining files, I pass them to echo with:

$ bargs #<buffer fling> echo
("GRIP.PY" "SWAM.PY" "")

I might actually do something like this with the function (notice the % character will be substituted with the list of files):

$ mkdir oddities

$ bargs #<buffer fling> mv % oddities

$ ls oddities

Bargs Code

Initial implementation of bargs:

(defun eshell/-buffer-as-args (buffer separator command)
  "Takes the contents of BUFFER, and splits it on SEPARATOR, and
runs the COMMAND with the contents as arguments. Use an argument
`%' to substitute the contents at a particular point, otherwise,
they are appended."
  (let* ((lines (with-current-buffer buffer
                   (buffer-substring-no-properties (point-min) (point-max))
         (subcmd (if (-contains? command "%")
                     (-flatten (-replace "%" lines command))
                   (-concat command lines)))
         (cmd-str  (string-join subcmd " ")))
    (message cmd-str)
    (eshell-command-result cmd-str)))

(defun eshell/bargs (buffer &rest command)
  "Passes the lines from BUFFER as arguments to COMMAND."
  (eshell/-buffer-as-args buffer "\n" command))

(defun eshell/sargs (buffer &rest command)
  "Passes the words from BUFFER as arguments to COMMAND."
  (eshell/-buffer-as-args buffer nil command))

EShell Summary

  • Advantages:
    • Similar shell experience between operating systems
    • Much more extendable, hackable and funner
  • Disadvantages:
    • Pipes go through Emacs buffers… not efficient
    • Programs that need special displays:

      (add-to-list 'eshell-visual-commands "top")

      For commands that have options that trigger curses/pager:

      (add-to-list 'eshell-visual-options '("git" "--help"))

      If command has a ncurses/pager sub-commands, use:

      (add-to-list 'eshell-visual-subcommands
         '("git" "log" "diff" "show"))

      Also set eshell-destroy-buffer-when-process-dies.

      My goal was to inspire potential hackery…