hamacs/ha-eshell.org

Expanding Eshell

A literate programming file for configuring the Emacs Shell.

Introduction

While I like vterm for logging into remote systems, I find Emacs’ shell, eshell, an interesting alternative. If you find the documentation lacking, I documented most features, and you might find the following helpful.

Navigation and Keys

Along with the regular Emacs keybindings, Eshell comes with some interesting features:

• M-RET gives you a prompt, even when you are running another command. Since eshell passes all input to subprocesses, there is no automatic input queueing as there is with other shells.
• C-c C-t truncates the buffer if it grows too large.
• C-c C-r will move point to the beginning of the output of the last command. With a prefix argument, eshell narrows to view its output.
• C-c C-o will delete the output from the last command.
• C-c C-f will move forward a complete shell argument.
• C-c C-b will move backward a complete shell argument.

Control-D Double Duty

Used to C-d exiting from a shell? Want it to keep working, but still allow deleting a character? We can have it both (thanks to wasamasa):

  (defun ha-eshell-quit-or-delete-char (arg)
    "The `C-d' sequence closes window or deletes a character."
    (interactive "p")
    (if (and (eolp) (looking-back eshell-prompt-regexp))
        (progn
          (eshell-life-is-too-much) ; Why not? (eshell/exit)
          (ignore-errors
            (delete-window)))
      (delete-forward-char arg)))

Pager Setup

If any program wants to pause the output through the $PAGER variable, well, we don't need that:

  (setenv "PAGER" "cat")

Argument Completion

Shell completion uses the flexible pcomplete mechanism internally, which allows you to program the completions per shell command. To know more, check out this blog post, about how to configure pcomplete for git commands. The pcmpl-args package extends pcomplete with completion support for more commands, like the Fish and other modern shells. I love how a package c→

  (use-package pcmpl-args)

Note that this will work with shell-command as well.

Better Command Line History

On this discussion a little gem for using IDO to search back through the history, instead of M-R to prompt for the history.

  (defun eshell-insert-history ()
    "Displays the eshell history to select and insert back into your eshell."
    (interactive)
    (insert (completing-read "Eshell history: "
                                 (delete-dups
                                  (ring-elements eshell-history-ring)))))

Predicate Filters and Modifiers

The T predicate filter allows me to limit file results that have internal org-mode tags. For instance, eshell will send files that have a #+TAGS: header with a mac label to the grep function:

  $ grep brew *.org(T'mac')

As described in this essay, to extend Eshell, we need a two-part function:

1. Parse the Eshell buffer to look for the parameter (and move the point past the parameter).
2. A predicate function that takes a file as a parameter.

For the first step, we have our function called as it helps parse the text. Based on what it sees, it returns the predicate function used to filter the files:

  (defun eshell-org-file-tags ()
    "Parse the eshell text at point.
  Looks for parameters surrounded in single quotes. Returns a
  function that takes a FILE and returns nil if the file given to
  it doesn't contain the org-mode #+TAGS: entry specified."

    ;; Step 1. Parse the eshell buffer for our tag between quotes
    ;;         Make sure to move point to the end of the match:
    (if (looking-at (rx "'" (group (one-or-more (not (or ")" "'"))))"'"))
        (let* ((tag (match-string 1))
               (reg (rx line-start
                        "#+" (optional "file") "tags:"
                        (one-or-more space)
                        (zero-or-more any)
                        (literal tag) word-end)))
          (goto-char (match-end 0))

          ;; Step 2. Return the predicate function:
          ;;         Careful when accessing the `reg' variable.
          `(lambda (file)
             (with-temp-buffer
               (insert-file-contents file)
               (re-search-forward ,reg nil t 1))))
      (error "The `T' predicate takes an org-mode tag value in single quotes.")))

Then we need add that function to the eshell-predicate-alist as the T tag:

  (defun ha-eshell-add-predicates ()
    "A hook to add a `eshell-org-file-tags' predicate filter to eshell."
    (add-to-list 'eshell-predicate-alist '(?T . (eshell-org-file-tags))))

Note: We can’t add it to the list until after we start our first eshell session, so we add it to the eshell-pred-load-hook.

Aliases

Gotta have some shell aliases, right? We have three ways of doing that. First, enter them into an eshell session:

  alias ll 'ls -AlohG --color=always'

Note that you need single quotes (not double quotes). Also note that more than one parameter doesn’t work with aliases (to resolve that, we need to write a function).

Second, you can create/populate the alias file, ~/.emacs.d/eshell/alias … as long as you don’t use those single quotes:

  alias ll ls -AlohG --color=always
  alias cls clear 1
  alias d dired $1
  alias find echo 'Please use fd <pattern> <paths> instead.'

Yeah, the variable $* doesn’t work as you’d expect, so use $1 when calling Emacs functions that take one parameter). For instance, while I would like to have the following, the real solution is to make functions (see below for details).

  alias less view-file

Third, you want more control, you can use the help:eshell/alias function, but it doesn’t honor $1 and other parameters, so we could create conditionally create function that we add to the eshell-mode-hook, for instance:

  (defun ha-eshell-add-aliases ()
    "Call `eshell/alias' to define my aliases."
    ;; The 'ls' executable requires the Gnu version on the Mac
    (let ((ls (if (file-exists-p "/usr/local/bin/gls")
                  "/usr/local/bin/gls"
                "/bin/ls")))
      (eshell/alias "ll" (concat ls " -AlohG --color=always"))))

I have also had a lot of trouble getting aliases to work, for instance dired works, but less does not:

  alias less view-file $1
  alias d dired $1

To work around this, I create functions instead.

Eshell Functions

Any function that begins with eshell/ is available as a command (with the remaining letters) Once I had a function eshell/f as a replacement for find, but the fd project is better.

Since eshell is an Emacs shell, I try to think how to use Emacs buffers in a shell-focused workflow. For instance, use view-file instead of less, as it will show a file with syntax coloring, and typing q returns to your shell session.

This helper function can tell me if an executable program is available, and return its location:

  (defun ha-find-executable (program)
    "Return full path to executable PROGRAM on the `exec-path'."
    (first
     (-filter 'file-executable-p
              (--map (expand-file-name program it) (exec-path)))))

Calling Emacs functions that take a single argument from eshell that could accept zero or more, can result in an error. This helper function can open each argument in a different window. It takes two functions, and calls the first function on the first argument, and calls the second function on each of the rest:

  (defun eshell-fn-on-files (fun1 fun2 args)
    "Call FUN1 on the first element in list, ARGS.
  Call FUN2 on all the rest of the elements in ARGS."
    (unless (null args)
      (let ((filenames (flatten-list args)))
        (funcall fun1 (car filenames))
        (when (cdr filenames)
          (mapcar fun2 (cdr filenames))))
      ;; Return an empty string, as the return value from `fun1'
      ;; probably isn't helpful to display in the `eshell' window.
      ""))

Less and More

While I can type find-file, I often use e as an alias for emacsclient in Terminals, so let’s do something similar for eshell: Also note that we can take advantage of the eshell-fn-on-files function to expand the find-file (which takes one argument), to open more than one file at one time.

  (defun eshell/e (&rest files)
    "Essentially an alias to the `find-file' function."
    (eshell-fn-on-files 'find-file 'find-file-other-window files))

  (defun eshell/ee (&rest files)
    "Edit one or more files in another window."
    (eshell-fn-on-files 'find-file-other-window 'find-file-other-window files))

No way would I accidentally type any of the following commands:

  (defalias 'eshell/emacs 'eshell/e)
  (defalias 'eshell/vi 'eshell/e)
  (defalias 'eshell/vim 'eshell/e)

Both less and more are the same to me. as I want to scroll through a file. Sure the bat program is cool, but from eshell, we could call view-file, and hit q to quit and return to the shell.

  (defun eshell/less (&rest files)
    "Essentially an alias to the `view-file' function."
    (eshell-fn-on-files 'view-file 'view-file-other-window files))

Do I type more any more than less?

  (defalias 'eshell/more 'eshell/less)
  (defalias 'eshell/view 'eshell/less)

Ebb and Flow output to Emacs Buffers

This is an interesting experiment.

Typing a command, but the output isn’t right. So you punch the up arrow, and re-run the command, but this time pass the output through executables like tr, grep, and even awk. Still not right? Rinse and repeat. Tedious. Since using Emacs to edit text is what we do best, what if we took the output of a command from Eshell, edit that output in a buffer, and then use that edited output in further commands?

I call this workflow of sending command output back and forth into an Emacs buffer, an ebb and flow approach, where the ebb function (for Edit a Bumped Buffer … or something like that), takes some command output, and opens it in a buffer (with an ebbflow minor mode), allowing us to edit or alter the data. Pull that data back to the Eshell session with the flow function (for Fetch buffer data by Lines or Words … naming is hard).

The ebbflow Buffer

If I don’t specify a specific buffer name, we use this default value:

  (defvar ha-eshell-ebbflow-buffername "*eshell-edit*"
    "The name of the buffer that eshell can use to store temporary input/output.")

This buffer has a minor-mode that binds C-c C-q to close the window and return to the Eshell that spawned it:

  (defun ha-eshell-ebbflow-return ()
    "Close the ebb-flow window and return to Eshell session."
    (interactive)
    (when (boundp 'ha-eshell-ebbflow-close-window)
      (bury-buffer))
    (when (boundp 'ha-eshell-ebbflow-return-buffer)
      (pop-to-buffer ha-eshell-ebbflow-return-buffer)))

  (define-minor-mode ebbflow-mode
    "Get your foos in the right places."
    :lighter " ebb"
    :keymap (let ((map (make-sparse-keymap)))
              (define-key map (kbd "C-c C-q") 'ha-eshell-ebbflow-return)
              map))

Since I use Evil, I also add Q to call this function:

  (evil-define-key 'normal ebbflow-mode-map (kbd "Q") 'ha-eshell-ebbflow-return)

Supporting Functions

I need a function to analyze command line options. I’ve tried to use eshell-eval-using-options, but it lacks the ability to have both dashed parameter arguments and non-parameter arguments. For instance, I want to type:

  flow --lines some-buffer another-buffer

To have both a —lines parameter, as well as a list of buffers, so I’ll need to roll my own. I will take some shortcuts. For instance, I will always have my double-dashed parameters start with the same letter, so we can have this predicate:

  (defun ha-eshell-parse-arg (argument dashed-letter)
    "Return true if ARGUMENT matches DASHED-LETTER."
    (when (stringp argument)
      (string-match (rx string-start "-" (optional "-") (literal dashed-letter)) argument)))

If an non-dashed parameter argument is of the wrong type (I’m looking for strings and buffers), the correct way to deal with that in Eshell is to call error with a message, so let’s wrap that functionality:

  (defun ha-eshell-arg-type-error (arg)
    "Throw an `error' about the incorrect type of ARG."
    (error (format "Illegal argument of type %s: %s\n%s"
                   (type-of arg) arg
                   (documentation 'eshell/flow))))

flow: (or Buffer Cat)

Eshell can send the output of a command sequence to a buffer:

  rg -i red > #<scratch>

But I can’t find a way to use the contents of buffers to use as part of the standard input to another as the start of a pipeline. Let’s create a function to fetch buffer contents.

I’m calling the ability to get a buffer contents, flow (Fetch contents as Lines Or Words). While this function will fetch the contents of any buffer, if one is not given, it will fetch the default, ha-eshell-ebbflow-buffername. Once the content is fetched, given the correct argument, it may convert the data:

as lines

separating the data on newlines. Useful for passing to for loops

as words

separating on spaces. Useful if the data is filenames

as a string

no conversion

  (defun eshell/flow (&rest args)
    "Output the contents of one or more buffers as a string.
  Usage: flow [OPTION] [BUFFER ...]
      -h, --help           show this usage screen
      -l, --lines          output contents as a list of lines
      -w, --words          output contents as a list of space-separated elements "
    (seq-let (conversion-type buffers) (eshell-flow-parse-args args)
      (let ((content-of-buffers (mapconcat 'eshell-flow-buffer-contents buffers "\n")))
        (cl-case conversion-type
          (:words (split-string content-of-buffers))
          (:lines (split-string content-of-buffers "\n"))
          (t      content-of-buffers)))))

Parsing the arguments is now hand-rolled:

  (defun eshell-flow-parse-args (args)
    "Parse ARGS and return a list of buffers and a format symbol."
    (let* ((first-arg (car args))
           (convert (cond
                     ;; Bugger out if our first argument is already a buffer ... no conversion needed
                     ((bufferp first-arg)        nil)
                     ;; No buffer option at all? Drop out as we'll get the default buffer:
                     ((null first-arg)           nil)
                     ;; Not a string or buffer, that is an error:
                     ((not (stringp first-arg)) (ha-eshell-arg-type-error first-arg))
                     ((ha-eshell-parse-arg first-arg "w")    :words)
                     ((ha-eshell-parse-arg first-arg "l")    :lines)
                     ((ha-eshell-parse-arg first-arg "h")    (error (documentation 'eshell/flow))))))
      (if convert
          (list convert (eshell-flow-buffers (cdr args)))
        (list nil (eshell-flow-buffers args)))))

Straight-forward to acquire the contents of a buffer :

  (defun eshell-flow-buffer-contents (buffer-name)
    "Return the contents of BUFFER as a string."
    (when buffer-name
      (save-window-excursion
        (switch-to-buffer buffer-name)
        (buffer-substring-no-properties (point-min) (point-max)))))

Specify the buffers with either the Eshell approach, e.g. #<buffer buffer-name>, or a string, ’*scratch*’, and if I don’t specify any buffer, we’ll use the default buffer:

  (defun eshell-flow-buffers (buffers)
    "Convert the list, BUFFERS, to actual buffers if given buffer names."
    (if buffers
        (--map (cond
              ((bufferp it) it)
              ((stringp it) (get-buffer it))
              (t            (ha-eshell-arg-type-error it)))
             buffers)
      ;; No buffers given? Use the default buffer:
      (list (get-buffer ha-eshell-ebbflow-buffername))))

I used to call this function, bcat (for buffer cat), and I sometimes type this:

  (defalias 'eshell/bcat 'eshell/flow)

ebb: Bump Data to a Buffer

We have three separate use-cases:

1. Execute a command, inserting the output into the buffer (good if we know the output will be long, complicated, or needing manipulation)
2. Insert one or more files into the buffer (this assumes the files are data)
3. Grab the output from the last executed Eshell command (what happens when we don’t give it a command string or files to read)

  (defun eshell/ebb (&rest args)
    "Run command with output into a buffer, or output of last command.
  Usage: ebb [OPTION] [COMMAND] [FILE ...]
      -h, --help           show this usage screen
      -a, --append         add command output to the *eshell-edit* buffer
      -p, --prepend        add command output to the end of *eshell-edit* buffer
      -i, --insert         add command output to *eshell-edit* at point"
    (seq-let (insert-location command files) (eshell-ebb-parse-args args)
      (cond
       (command (ha-eshell-ebb-command insert-location command))
       (files   (ha-eshell-ebb-files   insert-location files))
       (t       (ha-eshell-ebb-output  insert-location))))

    ;; At this point, we are in the `ha-eshell-ebbflow-buffername', and
    ;; the buffer contains the inserted data, so:
    (goto-char (point-min))

    nil) ; Return `nil' so that it doesn't print anything in `eshell'.

And we need an argument parser. I need to write a getopts clone or something similar, for while the following function works, it isn’t robust.

  (defun eshell-ebb-parse-args (args)
    "Parse ARGS and return a list of parts for the command.
  The elements of the returned list are:
    - location (symbol, either :insert :insert-location :prepend :replace)
    - string of commands
    - string of files

  The ARGS can have an optional argument, followed by
  a list of files, or a command list, or nothing."
      (let* ((first-arg (car args))
             (location (cond
                        ;; No buffer option at all? Drop out as we'll get the default buffer:
                        ((null first-arg)                    nil)
                        ((ha-eshell-parse-arg first-arg "i") :insert)
                        ((ha-eshell-parse-arg first-arg "a") :insert-location)
                        ((ha-eshell-parse-arg first-arg "p") :prepend)
                        ((ha-eshell-parse-arg first-arg "h") (error (documentation 'eshell/ebb))))))
        (if location
            (setq args (cdr args))
          (setq location :replace))
        (cond
         ((or (null args) (length= args 0)) (list location nil nil))
         ((file-exists-p (car args))        (list location nil args))
         (t                                 (list location args nil)))))

Each of the use-case functions described needs to switch to the *eshell-edit* buffer, and either clear it out or position the cursor.

  (defun ha-eshell-ebb-switch-to-buffer (insert-location)
    "Switch to `ha-eshell-ebbflow-buffername' and get the buffer ready for new data."
    (let ((return-buffer (current-buffer)))

      (if-let ((ebbwindow (get-buffer-window ha-eshell-ebbflow-buffername)))
          (select-window ebbwindow)
        (switch-to-buffer ha-eshell-ebbflow-buffername)
        (setq-local ha-eshell-ebbflow-close-window t))

      (setq-local ha-eshell-ebbflow-return-buffer return-buffer)
      (ebbflow-mode)

      (cl-case insert-location
        (:append  (goto-char (point-max)))
        (:prepend (goto-char (point-min)))
        (:insert   nil)
        (:replace (delete-region (point-min) (point-max))))))

Command string passed to eshell-command:

  (defun ha-eshell-ebb-command (insert-location command-parts)
    "Call `eshell-command' with the COMMAND-PARTS.
  Inserts the output into `ha-eshell-ebbflow-buffername'"
    (let ((command-string (string-join command-parts " ")))
      (ha-eshell-ebb-switch-to-buffer insert-location)
      (eshell-command command-string t)))

Given one or more filenames to the ebb command, concatenates each into the buffer.

  (defun ha-eshell-ebb-files (insert-location files)
    "Insert the FILES at the INSERT-LOCATION tin `ha-eshell-ebbflow-buffername'."
    (ha-eshell-ebb-switch-to-buffer insert-location)
    (dolist (file files)
      (insert-file file)
      (insert "\n")))

If we were not given a command to execute or a list of files to insert, we want to grab the output from the last executed command in the eshell buffer. To do this, we need to move to the start of the output, and then search for the prompt. Luckily Eshell assumes we have set up the eshell-prompt-regexp variable:

  (defun ha-eshell-ebb-output (insert-location)
    "Grab output from previous eshell command, inserting it into our buffer.
  Gives the INSERT-LOCATION to `ha-eshell-ebb-switch-to-buffer'."
    (let* ((start  (save-excursion
                     (goto-char eshell-last-output-start)
                     (re-search-backward eshell-prompt-regexp)
                     (next-line)
                     (line-beginning-position)))
           (end    eshell-last-output-start)
           (contents (buffer-substring-no-properties start end)))
      (ha-eshell-ebb-switch-to-buffer insert-location)
      (insert contents)))

Git

I used to have a number g-prefixed aliases to call git-related commands, but now, I call Magit instead. My gst command is an alias to magit-status, but using the alias doesn't pull in the current working directory, so I make it a function, instead:

  (defun eshell/gst (&rest args)
      (magit-status (pop args) nil)
      (eshell/echo))   ;; The echo command suppresses output

Replace ls

I like the output of the lsd program, and want ls to call it, if available.

  (defvar ha-lsd (ha-find-executable "lsd")
    "Location of the `lsd' program, if installed.")

The problem I have with lsd is that it does not display in columns or colorize its output in eshell (even when changing the TERM variable). Since I already wrote this code, I’m re-purposing it and expanding it. Step one is to have a function that gives a list of files for a directory (notice it doesn’t take options, for if I am going for special output, I’ll be calling ls directly).

  (defun ha-eshell-ls-files (&optional directory)
    "Return a list of directories in DIRECTORY or `default-directory' if null."
    (let ((default-directory (or directory default-directory)))
      (if ha-lsd
          (shell-command-to-list (format "%s --icon always" ha-lsd))

        (directory-files default-directory nil
                         (rx string-start
                             (not (any "." "#"))
                             (one-or-more any)
                             (not "~")
                             string-end)))))

Given a filename, let’s pad and colorize it based on file attributes:

  (defun ha-eshell-ls-filename (filename padded-fmt &optional directory)
    "Return a prettized version of FILE based on its attributes.
  Formats the string with PADDED-FMT."
    (let ((file (expand-file-name (if (string-match (rx (group alpha (zero-or-more any))) filename)
                                      (match-string 1 filename)
                                    filename)
                                  directory))
          (import-rx  (rx "README"))
          (image-rx   (rx "." (or "png" "jpg" "jpeg" "tif" "wav") string-end))
          (code-rx    (rx "." (or "el" "py" "rb") string-end))
          (docs-rx    (rx "." (or "org" "md") string-end)))
      (format padded-fmt
              (cond
               ((file-directory-p file)
                (propertize filename 'face 'eshell-ls-directory))
               ((file-executable-p file)
                (propertize filename 'face 'eshell-ls-executable))
               ((string-match import-rx file)
                (propertize filename 'face '(:foreground "orange")))
               ((string-match image-rx file)
                (propertize filename 'face 'eshell-ls-special))
               ((file-symlink-p file)
                (propertize filename 'face 'eshell-ls-symlink))
               ((not (file-readable-p file))
                (propertize filename 'face 'eshell-ls-unreadable))
               (t
                filename)))))

This function pulls all the calls to ha-eshell-ls-file to create columns to make a multi-line string:

  (defun ha-eshell-ls (&optional directory)
    "Return a formatted string of files for a directory.
  The string is a pretty version with columns and whatnot."
    (let* ((files   (ha-eshell-ls-files (or directory default-directory)))
           (longest (--reduce-from (max acc (length it)) 1 files))
           (width   (window-total-width))
           (columns (/ width (+ longest 3)))
           (padded  (if ha-lsd
                        (format "%%-%ds " longest)
                      (format "• %%-%ds " longest))))
      (cl-flet* ((process-lines (files)
                                (s-join "" (--map (ha-eshell-ls-filename it padded directory) files)))
                 (process-files (table)
                                (s-join "\n" (--map (process-lines it) table))))

        (concat (process-files (seq-partition files columns)) "\n\n"))))

While the ha-eshell-ls takes a directory, this version puts the canonical directory as a label before the listing, and this calls it directly specifying the directory name(s):

  (defun ha-eshell-ls-directory (directory)
    "Print the DIRECTORY name and its contents."
    (let ((dir (file-truename directory)))
      (concat
       (propertize dir 'face '(:foreground "gold" :underline t))
       ":\n"
       (ha-eshell-ls dir))))

I have the interface program to work with eshell.

  (defun eshell/lsd (&rest args)
    (let ((lsd (ha-find-executable "lsd")))
      (cond
       ;; I expect to call this function without any arguments most of the time:
       ((and lsd (null args))
        (ha-eshell-ls))
       ;; Called with other directories? Print them all, one at a time:
       ((and lsd (--none? (string-match (rx string-start "-") it) args))
        (mapconcat 'ha-eshell-ls-directory args ""))
       ;; Calling the function with -l or other arguments, don't bother. Call ls:
       (t (eshell/ls args)))))

Which needs an ls alias:

    ;; (eshell/alias "lss" "echo $@")

Regular Expressions

I think using the rx macro with applications like grep is great reason why eshell rocks. Assuming we can’t remember cryptic regular expression syntax, we could look for a GUID-like strings using ripgrep with:

  $ rg (rx (one-or-more hex) "-" (one-or-more hex))

The problem with this trick is that rx outputs an Emacs-compatible regular expression, which doesn’t always match regular expressions accepted by most applications.

The pcre2el project can convert from a Lisp regular expression to a PCRE (Perl Compatible Regular Expression), acceptable by ripgrep.

  (use-package pcre2el
    :straight (:host github :repo "joddie/pcre2el")
    :config
    (defmacro prx (&rest expressions)
      "Convert the rx-compatible regular EXPRESSIONS to PCRE.
    Most shell applications accept Perl Compatible Regular Expressions."
      `(rx-let ((integer (1+ digit))
                (float   (seq integer "." integer))
                (b256    (seq (optional (or "1" "2"))
                              (regexp "[0-9]\\{1,2\\}")))
                (ipaddr  (seq b256 "." b256 "." b256 "." b256))
                (time    (seq digit (optional digit) ":" (= 2 digit) (optional ":" (= 2 digit))))
                (date    (seq (= 2 digit) (or "/" "-") (= 2 digit) (or "/" "-") (= 4 digit)))
                (ymd     (seq (= 4 digit) (or "/" "-") (= 2 digit) (or "/" "-") (= 2 digit)))
                (uuid    (seq (= 8 hex) "-" (= 3 (seq (= 4 hex) "-")) (= 12 hex)))
                (guid    (seq uuid)))
         (rxt-elisp-to-pcre (rx ,@expressions)))))

Map

While I like eshell’s for loop well enough (if I can remember the syntax), as in:

  for file in *.org {
    chmod a+x $file
  }

I like the idea of using a map structure, for instance, wouldn’t it be cool to type something like:

  map chmod a+x *.org

How would this work without special syntax? Well, eshell sends the *.org as a list of files, which we could use as the delimiter. The downside is that we want to list the files, we need to actually list the files, as in:

  map chmod a+x (list "a.org" "c.org")

Pretty ugly, but what about using :: as a separator of the lambda from the list, like:

  map chmod a+x :: *.org b.txt

Here is my initial function. After separating the arguments into two groups (split on the :: string), we iterate over the file elements, creating a form that includes the filename.

  (defun eshell/map (&rest args)
    "Execute a command sequence over a collection of file elements.
  Separate the sequence and the elements with a `::' string.
  For instance:

      map chown _ angela :: *.org(u'oscar')

  The function substitutes the `_' sequence to a single filename
  element, and if not specified, it appends the file name to the
  command. So the following works as expected:

      map chmod a+x :: *.org"
    (seq-let (forms elements) (-split-on "::" args)
      (dolist (element (-flatten (-concat elements)))
        ;; Replace the _ or append the filename:
        (let* ((form (if (-contains? forms "_")
                         (-replace "_" element forms)
                       (-snoc forms element)))
               (cmd  (car form))
               (args (cdr form)))
          (eshell-named-command cmd args)))))

The eshell-named-command takes the command separately from the arguments, so we use car and cdr on the form.

Last Results

The shell-underscore project looks pretty cool, where the _ character represents a filename with the contents of the previous command (you know, like if you were planning on it, you’d tee at the end of every command). An interesting idea that I could duplicate.

While diving into the eshell source code, I noticed the special variables, $$ and $_ sometimes contains the output of the last command. For instance:

$ echo "hello world"
hello world
$ echo $$
hello world

What I would like is something like this to work:

$ ls *.org(U)
a.org b.org f.org
$ rg "foobar" $$

The problem may be between calling Emacs functions versus external commands, as the echo works, but the call to ls doesn’t:

$ ls *.org(U) b.txt
a.org b.org f.org b.txt

$ echo Nam $$
("Nam" nil)

I over-write that special variables to behave as expected:

• A hook runs after every command
• It copies the previous command’s output to a ring (so that I can get the last as well as the fifth one)
• Create a replacement function for $$ to read from my history ring

Let’s first make a ring that stores the output:

  (defvar ha-eshell-output (make-ring 10)
    "A ring (looped list) storing history of eshell command output.")

The following function does the work of saving the output of the last command. We can get this because after every command, eshell updates two variables, eshell-last-input-end (the start of the output), and eshell-last-output-start (the end of the output):

  (defun ha-eshell-store-last-output ()
    "Store the output from the last eshell command.
  Called after every command by connecting to the `eshell-post-command-hook'."
    (let ((output
           (buffer-substring-no-properties eshell-last-input-end eshell-last-output-start)))
      (ring-insert ha-eshell-output output)))

Now we save this output after every command by adding it to the eshell-post-command-hook:

  (add-hook 'eshell-post-command-hook 'ha-eshell-store-last-output)

Next, this function returns values from the history ring. I feel the need to have different ways of returning the output data. Unlike the behavior of the original shell (and most of its descendents, like bash), eshell doesn’t automatically split on whitespace. For instance, echo called this way:

$ echo a b *.txt
("a" "b"
 ("b.txt" "date today.txt"))

Given a list of three elements: a, b, and a list of all files in the current directory with an .org extension. An interesting side-effect is that spaces in filenames are often okay. If I specify and argument of text, it should return the command’s output as a string, but if I give it, list, it should contain the same information, but separated by spaces, into a list. For instance, if we are passing the output from ls to grep, we would use this format.

Like the shell-underscore project mentioned earlier, I can access the output stored from a file when given a file argument (the output will hold this temporary filename).

  (defun eshell/output (&rest args)
    "Return an eshell command output from its history.

  The first argument is the index into the historical past, where
  `0' is the most recent, `1' is the next oldest, etc.

  The second argument represents the returned output:
   ,* `text' :: as a string
   ,* `list' :: as a list of elements separated by whitespace
   ,* `file' :: as a filename that contains the output

  If the first argument is not a number, it assumes the format
  to be `:text'.
  "
    (let (frmt element)
      (cond
       ((> (length args) 1)  (setq frmt (cadr args)
                                   element (car args)))
       ((= (length args) 0)  (setq frmt "text"
                                   element 0))
       ((numberp (car args)) (setq frmt "text"
                                   element (car args)))
       ((= (length args) 1)  (setq frmt (car args)
                                   element 0)))

      (if-let ((results (ring-ref ha-eshell-output (or element 0))))
          (cl-case (string-to-char frmt)
            (?l     (split-string results))
            (?f     (ha-eshell-store-file-output results))
            (otherwise (s-trim results)))
        "")))

  (defun ha-eshell-store-file-output (results)
    "Writes the string, RESULTS, to a temporary file and returns that file name."
    (let ((filename (make-temp-file "ha-eshell-")))
      (with-temp-file filename
        (insert results))
      filename))

How would this function work in practice?

$ ls
a.org  b.txt  c.org  date today.txt  ever

$ output
a.org  b.txt  c.org  date today.txt  ever

$ echo { output list }
("a.org" "b.txt" "c.org" "date" "today.txt" "ever")

Notice how commands between { … } are eshell commands, otherwise, if I replace the braces with parens, I would have to write eshell/output. Let’s try the history feature:

$ echo "oldest"
oldest

$ echo "old"
old

$ echo "recent"
recent

$ echo "newest"
newest

$ echo { output 2 }
old

Eshell has a feature where special variables (stored in eshell-variable-aliases-list), can be a function. The $$ holds text-formatted output, and $_ contains list-formatted output, and $OUTPUT can be the output stored in a file.

  (with-eval-after-load "eshell"
    (defvar eshell-variable-aliases-list nil "Autoloading this eshell-defined variable")
    (add-to-list 'eshell-variable-aliases-list '("$"  ha-eshell-output-text))
    (add-to-list 'eshell-variable-aliases-list '("_"  ha-eshell-output-list))
    (add-to-list 'eshell-variable-aliases-list '("OUTPUT" ha-eshell-output-file)))

Without this change, the $$ variable calls eshell-last-command-result, where I believe my version (with history) may work more reliably. I define these helper functions:

  (defun ha-eshell-output (format-type indices)
    "Wrapper around `eshell/output' for the `eshell-variable-aliases-list'."
    (if indices
        (eshell/output (string-to-number (caar indices)) format-type)
      (eshell/output 0 format-type)))

  (defun ha-eshell-output-text (&optional indices &rest ignored)
    "A _text_ wrapper around `eshell/output' for the `eshell-variable-aliases-list'."
    (ha-eshell-output "text" indices))

  (defun ha-eshell-output-list (&optional indices &rest ignored)
    "A _list_ wrapper around `eshell/output' for the `eshell-variable-aliases-list'."
    (ha-eshell-output "list" indices))

  (defun ha-eshell-output-file (&optional indices &rest ignored)
    "A _file_ wrapper around `eshell/output' for the `eshell-variable-aliases-list'."
    (ha-eshell-output "file" indices))

How would this look? Something like:

$ echo a
a
$ echo b
b
$ echo c
c
$ echo $$
c
$ echo $$[2]
b

The final trick is being able to count backwards and remember they are always shifting. I guess if I wanted to remember the output for more than one command, I could do:

$ ls *.org(U) b.txt
a.org  b.txt

$ chmod o+w $_

$ rg Nam $_[1]
a.org
8:Nam vestibulum accumsan nisl.

b.txt
1:Nam euismod tellus id erat.
7:Name three animals that start with C

Wanna see something cool about Eshell? Let’s swirl Lisp and Shell commands:

$ rg (rx line-start "Nam ") $_[2]
b.txt
1:Nam euismod tellus id erat.

a.org
8:Nam vestibulum accumsan nisl.

Special Prompt

Following these instructions, we build a better prompt with the Git branch in it (Of course, it matches my Bash prompt). First, we need a function that returns a string with the Git branch in it, e.g. ":master"

  (defun curr-dir-git-branch-string (pwd)
    "Returns current git branch as a string, or the empty string if
  PWD is not in a git repo (or the git command is not found)."
    (interactive)
    (when (and (not (file-remote-p pwd))
               (eshell-search-path "git")
               (locate-dominating-file pwd ".git"))
      (let* ((git-url    (shell-command-to-string "git config --get remote.origin.url"))
             (git-repo   (file-name-base (s-trim git-url)))
             (git-output (shell-command-to-string (concat "git rev-parse --abbrev-ref HEAD")))
             (git-branch (s-trim git-output))
             (git-icon   "\xe0a0")
             (git-icon2  (propertize "\xf020" 'face `(:family "octicons"))))
        (concat git-repo " " git-icon2 " " git-branch))))

The function takes the current directory passed in via pwd and replaces the $HOME part with a tilde. I'm sure this function already exists in the eshell source, but I didn't find it…

  (defun pwd-replace-home (pwd)
    "Replace home in PWD with tilde (~) character."
    (interactive)
    (let* ((home (expand-file-name (getenv "HOME")))
           (home-len (length home)))
      (if (and
           (>= (length pwd) home-len)
           (equal home (substring pwd 0 home-len)))
          (concat "~" (substring pwd home-len))
        pwd)))

Make the directory name be shorter… by replacing all directory names with its first names. We leave the last two to be the full names. Why yes, I did steal this.

  (defun pwd-shorten-dirs (pwd)
    "Shorten all directory names in PWD except the last two."
    (let ((p-lst (split-string pwd "/")))
      (if (> (length p-lst) 2)
          (concat
           (mapconcat (lambda (elm) (if (zerop (length elm)) ""
                                 (substring elm 0 1)))
                      (butlast p-lst 2)
                      "/")
           "/"
           (mapconcat (lambda (elm) elm)
                      (last p-lst 2)
                      "/"))
        pwd)))  ;; Otherwise, we return the PWD

Break up the directory into a "parent" and a "base":

  (defun split-directory-prompt (directory)
    (if (string-match-p ".*/.*" directory)
        (list (file-name-directory directory) (file-name-base directory))
      (list "" directory)))

Using virtual environments for certain languages is helpful to know, since I change them based on the directory.

  (defun ruby-prompt ()
    "Returns a string (may be empty) based on the current Ruby Virtual Environment."
    (let* ((executable "~/.rvm/bin/rvm-prompt")
           (command    (concat executable "v g")))
      (when (file-exists-p executable)
        (let* ((results (shell-command-to-string executable))
               (cleaned (string-trim results))
               (gem     (propertize "\xe92b" 'face `(:family "alltheicons"))))
          (when (and cleaned (not (equal cleaned "")))
            (s-replace "ruby-" gem cleaned))))))

  (defun python-prompt ()
    "Returns a string (may be empty) based on the current Python
     Virtual Environment. Assuming I've called the M-x command:
     `pyenv-mode-set'."
    (when (fboundp #'pyenv-mode-version)
      (let ((venv (pyenv-mode-version)))
        (when venv
          (concat
           (propertize "\xe928" 'face `(:family "alltheicons"))
           (pyenv-mode-version))))))

Now tie it all together with a prompt function can color each of the prompts components.

  (defun eshell/eshell-local-prompt-function ()
    "A prompt for eshell that works locally (in that it assumes it
  could run certain commands) to make a prettier, more-helpful
  local prompt."
    (interactive)
    (let* ((pwd        (eshell/pwd))
           (directory (split-directory-prompt
                       (pwd-shorten-dirs
                        (pwd-replace-home pwd))))
           (parent (car directory))
           (name   (cadr directory))
           (branch (curr-dir-git-branch-string pwd))
           (ruby   (when (not (file-remote-p pwd)) (ruby-prompt)))
           (python (when (not (file-remote-p pwd)) (python-prompt)))

           (dark-env (eq 'dark (frame-parameter nil 'background-mode)))
           (for-bars                 `(:weight bold))
           (for-parent  (if dark-env `(:foreground "dark orange") `(:foreground "blue")))
           (for-dir     (if dark-env `(:foreground "orange" :weight bold)
                          `(:foreground "blue" :weight bold)))
           (for-git                  `(:foreground "green"))
           (for-ruby                 `(:foreground "red"))
           (for-python               `(:foreground "#5555FF")))

      (concat
       (propertize "⟣─ "    'face for-bars)
       (propertize parent   'face for-parent)
       (propertize name     'face for-dir)
       (when branch
         (concat (propertize " ── "    'face for-bars)
                 (propertize branch   'face for-git)))
       ;; (when ruby
       ;;   (concat (propertize " ── " 'face for-bars)
       ;;           (propertize ruby   'face for-ruby)))
       ;; (when python
       ;;   (concat (propertize " ── " 'face for-bars)
       ;;           (propertize python 'face for-python)))
       (propertize "\n"     'face for-bars)
       (propertize (if (= (user-uid) 0) " #" " $") 'face `(:weight ultra-bold))
       ;; (propertize " └→" 'face (if (= (user-uid) 0) `(:weight ultra-bold :foreground "red") `(:weight ultra-bold)))
       (propertize " "    'face `(:weight bold)))))

  (setq-default eshell-prompt-function #'eshell/eshell-local-prompt-function)

Here is the result:

Simple Prompt with Mode Line

To achieve more screen estate, leave your prompt simple:

  (setq eshell-prompt-function (lambda () "$ "))

Display detailed information, like the current working directory, in the mode line using which-function-mode.

The eshell/pwd function returns the current working directory, but we need to have a function that returns that only in eshell-mode, otherwise, we will have the current working directory in every buffer:

  (defun ha-eshell-mode-line ()
    "Return the current working directory if in eshell-mode."
    (when (eq major-mode 'eshell-mode)
      (thread-last default-directory
                   (s-replace-regexp (rx (eval (getenv "HOME"))) "~")
                   (s-replace-regexp (rx "/" line-end) ""))))

Add this function to the which-func-functions list:

  (add-to-list 'which-func-functions 'ha-eshell-mode-line)

Turn on the global minor mode to display this. See Toggle Switches leader for that.

Fringe Status

The eshell-fringe-status project shows a color-coded icon of the previous command run (green for success, red for error). Doesn’t work reliably, but the fringe is inconspicuous. Seems to me, that if would be useful to rejuggle those fringe markers so that the marker matched the command entered (instead of seeing a red mark, and needing to scroll back to seethe command that made the error). Still…

  (use-package eshell-fringe-status
    :hook (eshell-mode . eshell-fringe-status-mode))

Opening Banner

Whenever I open a shell, I instinctively type ls … so why not do that automatically? The eshell-banner-message variable, while defaults to a string, this variable can be a form (an s-expression) that calls a function, so I made a customized ls that can be attractive:

  (defun ha-eshell-banner ()
    "Return a string containing the files in the current directory."
    (eshell/lsd))

Shell Windows

Now that I often need to pop into remote systems to run a shell or commands, I create helper functions to create those buffer windows. Each buffer begins with eshell: allowing me to have more than one eshells, typically, one per project.

Shell There

The basis for distinguishing a shell is its parent location. Before starting eshell, we make a small window, set the buffer name (using the eshell-buffer-name):

  (defun eshell-there (parent)
    "Open an eshell session in a PARENT directory.
  The window is smaller and named after this directory."
    (let* ((name (thread-first parent
                               (split-string "/" t)
                               (last)
                               (car)))
           (height (/ (window-total-height) 3))
           (default-directory parent))
      (split-window-vertically (- height))
      (setq eshell-buffer-name (format "*eshell: %s*" name))
      (eshell)))

Shell Here

This version of the eshell bases the location on the current buffer’s parent directory:

  (defun eshell-here ()
    "Opens a new shell in the directory of the current buffer.
  Renames the eshell buffer to match that directory to allow more
  than one eshell window."
    (interactive)
    (eshell-there (if (buffer-file-name)
                      (file-name-directory (buffer-file-name))
                    default-directory)))

And let’s bind it:

  (bind-key "C-!" 'eshell-here)

Shell for a Project

This version starts eshell in the project’s root, using projectile-project-root:

  (defun eshell-project ()
    "Open a new shell in the project root directory, in a smaller window."
      (interactive)
      (eshell-there (projectile-project-root)))

And we can attach this function to the projectile menu:

  (ha-leader "p t" '("eshell" . eshell-project))

Shell Over There

Would be nice to be able to run an eshell session and use Tramp to connect to the remote host in one fell swoop:

  (defun eshell-remote (host)
    "Creates an eshell session that uses Tramp to automatically
  connect to a remote system, HOST.  The hostname can be either the
  IP address, or FQDN, and can specify the user account, as in
  root@blah.com. HOST can also be a complete Tramp reference."
    (interactive "sHost: ")

    (let ((destination-path
           (cond
            ;; Is the HOST already an absolute tramp reference?
            ((string-match-p (rx line-start "/") host) host)

            ;; Does it match any acceptable reference? Get the parts:
            ((string-match-p (ha-eshell-host-regexp 'full) host)
             (string-match (ha-eshell-host-regexp 'full) host) ;; Why!?
             (let* ((user1 (match-string 2 host))
                    (host1 (match-string 3 host))
                    (user2 (match-string 6 host))
                    (host2 (match-string 7 host)))
               (if host1
                   (ha-eshell-host->tramp user1 host1)
                 (ha-eshell-host->tramp user2 host2))))

            ;; Otherwise, we assume we have a hostname from a string?
            ;; Convert to a simple 'default' tramp URL:
            (t (format "/%s:" host)))))
      (eshell-there destination-path)))

Shell Here to There

Since I have Org files that contains tables of system to remotely connect to, I figured I should have a little function that can jump to a host found listed anywhere on the line.

The regular expression associated with IP addresses, hostnames, user accounts (of the form, jenkins@my.build.server, or even full Tramp references, is a bit…uhm, hairy. And since I want to reuse these, I will hide them in a function:

  (defun ha-eshell-host-regexp (regexp)
    "Returns a particular regular expression based on symbol, REGEXP"
    (let* ((user-regexp      "\\(\\([[:alnum:]._-]+\\)@\\)?")
           (tramp-regexp     "\\b/ssh:[:graph:]+")
           (ip-char          "[[:digit:]]")
           (ip-plus-period   (concat ip-char "+" "\\."))
           (ip-regexp        (concat "\\(\\(" ip-plus-period "\\)\\{3\\}" ip-char "+\\)"))
           (host-char        "[[:alpha:][:digit:]-]")
           (host-plus-period (concat host-char "+" "\\."))
           (host-regexp      (concat "\\(\\(" host-plus-period "\\)+" host-char "+\\)"))
           (horrific-regexp  (concat "\\b"
                                     user-regexp ip-regexp
                                     "\\|"
                                     user-regexp host-regexp
                                     "\\b")))
      (cond
       ((eq regexp 'tramp) tramp-regexp)
       ((eq regexp 'host)  host-regexp)
       ((eq regexp 'full)  horrific-regexp))))

The function to scan a line for hostname patterns uses different function calls that what I could use for eshell-there, so let's save-excursion and hunt around:

  (defun ha-eshell-scan-for-hostnames ()
    "Helper function to scan the current line for any hostnames, IP
  or Tramp references.  This returns a tuple of the username (if
  found) and the hostname.

  If found a Tramp reference, the username part of the tuple is `nil'."
    (save-excursion
      (goto-char (line-beginning-position))
      (if (search-forward-regexp (ha-eshell-host-regexp 'tramp) (line-end-position) t)
          (cons nil (buffer-substring-no-properties (match-beginning 0) (match-end 0)))

        ;; Returns the text associated with match expression, NUM or `nil' if found no match
        (cl-flet ((ha-eshell-get-expression (num) (if-let ((first (match-beginning num))
                                                           (end   (match-end num)))
                                                      (buffer-substring-no-properties first end))))

          (search-forward-regexp (ha-eshell-host-regexp 'full) (line-end-position))

          ;; Until robust, let's keep this debugging code here:
          ;; (message (mapconcat (lambda (tup) (if-let ((s (car tup))
          ;;                                       (e (cadr tup)))
          ;;                                  (buffer-substring-no-properties s e)
          ;;                                "null"))
          ;;             (-partition 2 (match-data t)) " -- "))

          (let ((user1 (ha-eshell-get-expression 2))
                (host1 (ha-eshell-get-expression 3))
                (user2 (ha-eshell-get-expression 6))
                (host2 (ha-eshell-get-expression 7)))
            (if host1
                (list user1 host1)
              (list user2 host2)))))))

Tramp reference can be long when attempting to connect as another user account using the pipe symbol.

  (defun ha-eshell-host->tramp (username hostname &optional prefer-root)
    "Return a TRAMP reference based on a USERNAME and HOSTNAME
  that refers to any host or IP address."
    (cond ((string-match-p "^/" host)
             host)
          ((or (and prefer-root (not username)) (equal username "root"))
             (format "/ssh:%s|sudo:%s:" hostname hostname))
          ((or (null username) (equal username user-login-name))
             (format "/ssh:%s:" hostname))
          (t
             (format "/ssh:%s|sudo:%s|sudo@%s:%s:" hostname hostname username hostname))))

This function pulls it all together:

  (defun eshell-here-on-line (p)
    "Search the current line for an IP address or hostname, and call the `eshell-here' function.

  Call with PREFIX to connect with the `root' useraccount, via
  `sudo'."
    (interactive "p")
    (seq-let (user host) (ha-eshell-scan-for-hostnames)
      (let ((destination (ha-eshell-host->tramp user host (> p 1))))
        (message "Connecting to: %s" destination)
        (eshell-there destination))))

Command on the File Buffer

Sometimes you need to change something about the current file you are editing…like the permissions or even execute it. Hitting Command-1 will prompt for a shell command string and then append the current file to it and execute it.

  (defun execute-command-on-file-buffer (cmd)
    "Executes a shell command, CMD, on the current buffer's file.
  Appends the filename to the command if not specified, so:

      chmod a+x

  Works as expected. We replace the special variable `$$' with the
  filename of the buffer. Note that `eshell-command' executes this
  command, so eshell modifiers are available, for instance:

      mv $$ $$(:r).txt

  Will rename the current file to now have a .txt extension.
  See `eshell-display-modifier-help' for details on that."

    (let* ((file-name (buffer-file-name))
           (full-cmd (cond ((string-match (rx "$$") cmd)
                            (replace-regexp-in-string (rx "$$") file-name cmd))
                           ((and file-name (string-match (rx (literal file-name)) cmd))
                            cmd)
                           (t
                            (concat cmd " " file-name)))))
      (message "Executing: %s" full-cmd)
      (eshell-command full-cmd)))

Configuration

Here is where we associate all the functions and their hooks with eshell, through the magic of use-package.

  (use-package eshell
    :straight (:type built-in)
    :custom (eshell-banner-message '(ha-eshell-banner))

    :init
    (setq eshell-error-if-no-glob t
          ;; This jumps back to the prompt:
          eshell-scroll-to-bottom-on-input 'all
          eshell-hist-ignoredups t
          eshell-save-history-on-exit t

          ;; Since eshell starts fast, let's dismiss it on exit:
          eshell-kill-on-exit t
          eshell-destroy-buffer-when-process-dies t

          ;; Can you remember the parameter differences between the
          ;; executables `chmod' and `find' and their Emacs counterpart?
          ;; Me neither, so this makes it act a bit more shell-like:
          eshell-prefer-lisp-functions nil)

    :hook ((eshell-pred-load . ha-eshell-add-predicates))

    :bind (("M-!" . eshell-command)
           ("s-1" . execute-command-on-file-buffer)
           :map eshell-mode-map
           ("M-R"   . eshell-insert-history)
           ("C-d"   . ha-eshell-quit-or-delete-char)))

Note that the default list to eshell-visual-commands is good enough.

Add leader commands to call my defined functions:

  (ha-leader
    "!" '("eshell cmd" . execute-command-on-file-buffer)
    "a e"   '(:ignore t :which-key "eshell")
    "a e e" '("new eshell"          . eshell-here)
    "a e r" '("remote"              . eshell-remote)
    "a e p" '("project"             . eshell-project)
    "a e g" '("at point"            . eshell-here-on-line)
    "a e b" '("exec on file-buffer" . execute-command-on-file-buffer))

No, i’m not sure why use-package has an issue with both :hook, :bind and :config directives in sequence.