Remote Access to Systems

Remote Editing with Tramp
Remote Terminals
Keybindings

A literate configuration for accessing remote systems.

Remote Editing with Tramp

Tramp allows almost all Emacs features to execute on a remote system.

(use-package tramp
  :straight (:type built-in)

  :config
  ;; Use remote PATH on tramp (handy for eshell).
  (add-to-list 'tramp-remote-path 'tramp-own-remote-path)

  ;; Make sure version control system doesn't slow tramp:
  (setq vc-ignore-dir-regexp
        (format "%s\\|%s" vc-ignore-dir-regexp tramp-file-name-regexp)))

Will Schenk has a simple extension to allow editing of files inside a Docker container:

(use-package tramp
  :straight (:type built-in)
  :config
  (push '("docker" . ((tramp-login-program "docker")
                      (tramp-login-args (("exec" "-it") ("%h") ("/bin/sh")))
                      (tramp-remote-shell "/bin/sh")
                      (tramp-remote-shell-args ("-i") ("-c"))))
        tramp-methods)

  (defadvice tramp-completion-handle-file-name-all-completions
      (around dotemacs-completion-docker activate)
    "(tramp-completion-handle-file-name-all-completions \"\" \"/docker:\" returns
    a list of active Docker container names, followed by colons."
    (if (equal (ad-get-arg 1) "/docker:")
        (let* ((command "docker ps --format '{{.Names}}:'")
               (dockernames-raw (shell-command-to-string command))
               (dockernames (split-string dockernames-raw "\n")))
          (setq ad-return-value dockernames))
      ad-do-it)))

Keep in mind you need to name your Docker session, with the —name option. I actually do more docker work on remote systems (as Docker seems to make my fans levitate my laptop over the desk). Granted, the URL is a bit lengthy, for instance:

/ssh:kolla-compute1.cedev13.d501.eng.pdx.wd|sudo:kolla-compute1.cedev13.d501.eng.pdx.wd|docker:kolla_toolbox:/

Which means, I need to put it as a link in an org file.

Note: That we need to have Tramp SSH option comes from my personal .ssh/config file instead of its internal cache:

(use-package tramp-sh
  :after tramp
  :straight (:type built-in)
  :custom (tramp-use-ssh-controlmaster-options nil))

Remote Terminals

Sure iTerm is nice for connecting and running commands on remote systems, however, it lacks a command line option that allows you to select and manipulate the displayed text without a mouse. This is where Emacs can shine.

Feature One:

When calling the ha-ssh function, it opens a vterm window which, unlike other terminal emulators in Emacs, merges both Emacs and Terminal behaviors. Essentially, it just works. It vterm isn’t installed, it falls back to term.

Preload a list of favorite/special hostnames with multiple calls to:

(ha-ssh-add-favorite-host "Devbox 42" "10.0.1.42")

Then calling ha-ssh function, a list of hostnames is available to quickly jump on a system (with the possibility of fuzzy matching if you have Helm or Ivy installed).

This also has the ability to call OpenStack to gather the hostnames of dynamic systems (what I call “an Overcloud”), which is appended to the list of favorite hostnames. The call to OpenStack only needs to be called once, since the hosts are then cached, see ha-ssh-overcloud-query-for-hosts.

Feature Two:

Use the favorite host list to quickly edit a file on a remote system using Tramp, by calling either ha-ssh-find-file and ha-ssh-find-root.

Feature Three:

Working with remote shell connections programmatically, for instance:

(let ((win-name "some-host"))
  (ha-ssh "some-host.in.some.place" win-name)
  (ha-ssh-send "source ~/.bash_profile" win-name)
  (ha-ssh-send "clear" win-name))
;; ...
(ha-ssh-exit win-name)

Actually the win-name in this case is optional, as it will use a good default.

VTerm

I’m not giving up on Eshell, but I am playing around with vterm, and it is pretty good, but I use it primarily as a more reliable approach for remote terminal sessions.

VTerm has an issue (at least for me) with M-Backspace not deleting the previous word, and yeah, I want to make sure that both keystrokes do the same thing.

(use-package vterm
  :config
  (dolist (k '("<C-backspace>" "<M-backspace>"))
    (define-key vterm-mode-map (kbd k)
                (lambda () (interactive) (vterm-send-key (kbd "C-w")))))

  ;; Enter copy mode? Go to Evil's normal state to move around:
  (when (fboundp 'evil-normal-state)
    (advice-add 'vterm-copy-mode :after 'evil-normal-state))

  :hook
  (vterm-mode . (lambda ()
                  (when (boundp 'evil-insert-state-cursor)
                    (setq-local evil-insert-state-cursor 'box))
                  (setq-local show-paren-mode nil)
                  (setf truncate-lines nil
                        vterm-use-vterm-prompt-detection-method nil
                        term-prompt-regexp "^.* $ ")
                  (flycheck-mode -1)
                  (yas-minor-mode -1) ;; This actually code be interesting, but...

                  (when (fboundp 'evil-insert-state)
                    (evil-insert-state)))))

The advantage of running terminals in Emacs is the ability to copy text without a mouse. For that, hit C-c C-t to enter a special copy-mode. If I go into this mode, I might as well also go into normal mode to move the cursor. To exit the copy-mode (and copy the selected text to the clipboard), hit Return.

Variables

Let’s begin by defining some variables used for communication between the functions.

(defvar ha-latest-ssh-window-name nil
  "The window-name of the latest ssh session. Most commands default to the last session.")

(defvar ha-ssh-host-history '() "List of hostnames we've previously connected.")

(defvar ha-ssh-favorite-hostnames '()
  "A list of tuples (associate list) containing a hostname and its IP address.
See =ha-ssh-add-favorite-host= for easily adding to this list.")

Also, let’s make it easy for me to change my default shell:

(defvar ha-ssh-shell (shell-command-to-string "type -p fish")
  "The executable to the shell I want to use locally.")

Interactive Interface to Remote Systems

The function, ha-ssh pops up a list of favorite hosts and then uses the vterm functions to automatically SSH into the chosen host:

(defun ha-ssh (hostname &optional window-name)
  "Start a SSH session to a given HOSTNAME (with an optionally specified WINDOW-NAME).
If called interactively, it presents the user with a list
returned by =ha-ssh-choose-host=."
  (interactive (list (ha-ssh-choose-host)))
  (unless window-name
    (setq window-name (format "ssh: %s" hostname)))
  (setq ha-latest-ssh-window-name (format "*%s*" window-name))

  ;; I really like this =vterm= interface, so if I've got it loaded, let's use it:
  (if (not (fboundp 'vterm))
      ;; Should we assume the =ssh= we want is on the PATH that started Emacs?
      (make-term window-name "ssh" nil hostname)
    (vterm ha-latest-ssh-window-name)
    (vterm-send-string (format "ssh %s" hostname))
    (vterm-send-return))

  (pop-to-buffer ha-latest-ssh-window-name))

Of course, we need a function that interactive can call to get that list, and my thought is to call helm if it is available, otherwise, assume that ido/ivy will take over the completing-read function:

(defun ha-ssh-choose-host ()
  "Prompts the user for a host, and if it is in the cache, return
its IP address, otherwise, return the input given.
This is used in calls to =interactive= to select a host."
  (let ((hostname
         ;; We call Helm directly if installed, only so that we can get better
         ;; labels in the window, otherwise, the =completing-read= call would be fine.
         (if (fboundp 'helm-comp-read)
             (helm-comp-read "Hostname: " ha-ssh-favorite-hostnames
                             :name "Hosts"
                             :fuzzy t :history ha-ssh-host-history)
           (completing-read "Hostname: " ha-ssh-favorite-hostnames nil 'confirm nil 'ha-ssh-host-history))))
    (alist-get hostname ha-ssh-favorite-hostnames hostname nil 'equal)))

Simply calling vterm fails to load my full environment, so this allows me to start the terminal in a particular directory (defaulting to the root of the current project):

(defun ha-shell (&optional directory name)
  "Creates and tidies up a =vterm= terminal shell in side window."
  (interactive (list (read-directory-name "Starting Directory: " (project-root (project-current)))))
  (let* ((win-name (or name (ha-shell--name-from-dir directory)))
         (buf-name (format "*%s*" win-name))
         (default-directory (or directory default-directory)))
    (setq ha-latest-ssh-window-name buf-name)
    (if (not (fboundp 'vterm))
        (make-term win-name ha-ssh-shell)
      (vterm buf-name))))

Before we leave this section, I realize that I would like a way to add to my list of hosts:

(defun ha-ssh-add-favorite-host (hostname ip-address)
  "Add a favorite host to your list for easy pickin's."
  (interactive "sHostname: \nsIP Address: ")
  (add-to-list 'ha-ssh-favorite-hostnames (cons hostname ip-address)))

Let’s have a quick way to bugger out of the terminal:

(defun ha-ssh-exit (&optional window-name)
  "End the SSH session specified by WINDOW-NAME (or if not, the latest session)."
  (interactive)
  (unless (string-match-p "v?term" (buffer-name))
    (unless window-name
      (setq window-name ha-latest-ssh-window-name))
    (pop-to-buffer window-name))

  (ignore-errors
    (term-send-eof))
  (kill-buffer window-name)
  (delete-window))

Programmatic Interface

Now that Emacs can host a Terminal shell, I would like to programmatically send commands to the running terminal, e.g. (ha-shell-send "ls *.py")

Since every project perspective may have a shell terminal, let’s see if I can figure which shell buffer to send—based on the current-directory.

(defun ha-shell-send (command &optional directory)
  "Send COMMAND to existing shell terminal based on DIRECTORY.
If the shell doesn't already exist, start on up by calling
the `ha-shell' function.

The real work for this is done by `ha-ssh-send'.

If DIRECTORY is nil, use the project root from project."
  (let ((buf (ha-shell--buf-from-dir directory)))
    (unless buf
      (setq buf (ha-shell directory)))
    (ha-ssh-send command buf)))

(defun ha-shell--buf-from-dir (directory)
  "Return Terminal buffer associated with DIRECTORY.
Or nil if no buffer has been found."
  (let* ((win-name (ha-shell--name-from-dir directory))
         (win-rx   (rx "*" (literal win-name) "*"))
         (bufs     (seq-filter (lambda (b) (when (string-match win-rx (buffer-name b)) b))
                               (buffer-list))))
     (first bufs)))

(defun ha-shell--name-from-dir (&optional directory)
  "Return an appropriate title for a terminal based on DIRECTORY.
If DIRECTORY is nil, use the `project-name'."
  (unless directory
    (setq directory (project-name (project-current))))
  (let ((name
         ;; Most of the time I just want the base project name, but in
         ;; my "work" directory, the projects are too similar, and I
         ;; need two levels of directories to distinguish them as a
         ;; project.
         (if (s-contains? "/work/" directory)
             (thread-last directory
                          (s-split "/")
                          (-remove 's-blank-str?)
                          (-take-last 2)
                          (s-join "/"))
           (file-name-base (directory-file-name directory)))))
    (format "Terminal: %s" name)))

Perhaps a Unit test is in order:

(ert-deftest ha--terminal-name-from-dir-test ()
  (should
   (string= (ha-shell--name-from-dir "~/src/hamacs/") "Terminal: hamacs"))
  (should
   (string= (ha-shell--name-from-dir "~/work/foo/bar") "Terminal: foo/bar"))
  (should
   (string= (ha-shell--name-from-dir) "Terminal: hamacs")))

The previous functions (as well as my own end of sprint demonstrations) often need to issue some commands to a running terminal session, which is a simple wrapper around a send text and send return sequence:

(defun ha-ssh-send (phrase &optional window-name)
  "Send command PHRASE to the currently running SSH instance.
If you want to refer to another session, specify the correct WINDOW-NAME.
This is really useful for scripts and demonstrations."
  (unless window-name
    (setq window-name ha-latest-ssh-window-name))
  (save-window-excursion
    (pop-to-buffer window-name)

    (if (fboundp 'vterm)
        (progn
          (vterm-send-string phrase)
          (vterm-send-return))
      (progn
        (term-send-raw-string phrase)
        (term-send-input)))))

As you may know, I’m big into literate devops where I put my shell commands in org files. However, I also work as part of a team that for some reason, doesn’t accept Emacs as their One True Editor. At least, I am able to talk them into describing commands in Markdown files, e.g. README.md. Instead of copying-pasting into the shell, could I send the current command to that shell?

(defun ha-ssh-send-line (prefix)
  "Copy the contents of the current line in the current buffer,
and call =ha-ssh-send= with it. After sending the contents, it
returns to the current line."
  (interactive "P")
  ;; The function =save-excursion= doesn't seem to work...
  (let ((buf (current-buffer)))
    (dolist (line (ha-ssh--line-or-block prefix))
      ;; (sit-for 0.25)
      (ha-ssh-send line))
    (pop-to-buffer buf)))

What does current command mean? The current line? A good fall back. Selected region? Sure, if active, but that seems like more work. In a Markdown file, I can gather the entire source code block, just like in an Org file. So the following function may be a bit complicated in determining what is this current code:

(defun ha-ssh--line-or-block (num-lines)
  "Return a list of the NUM-LINES from current buffer.
If NUM-LINES is nil, then follow these rules:
If the region is active, return the lines from that.
If in an org-mode block, return that block.
If in a Markdown file, return the triple-back-tick code,
or the indented code, or the inline code between single ticks.
Otherwise, just return the current line."
  (ha-ssh--line-cleanup
   (cond
    ((and num-lines (numberp num-lines))
     (buffer-substring-no-properties
      (line-beginning-position) (line-end-position num-lines)))

    ;; Region active?
    ((region-active-p)
     (buffer-substring-no-properties
      (region-beginning) (region-end)))

    ;; In org? Use the block
    ((and (eq major-mode 'org-mode) (org-in-src-block-p))
     (org-element-property :value (org-element-at-point)))

    ;; In Markdown block?
    ((and (eq major-mode 'markdown-mode) (markdown-code-block-at-point-p))
     (buffer-substring-no-properties
      (car (markdown-code-block-at-point-p))
      (cadr (markdown-code-block-at-point-p))))

    ;; In Markdown code that is just on part of the line?
    ((and (eq major-mode 'markdown-mode) (markdown-inline-code-at-point-p))
     (buffer-substring-no-properties
      (car (markdown-inline-code-at-point-p))
      (cadr (markdown-inline-code-at-point-p))))

    (t      ; Otherwise, just grab the current line:
     (buffer-substring-no-properties
      (line-beginning-position) (line-end-position))))))

In Markdown (and org), I might have initial spaces that should be removed (but not all initial spaces):

(defun ha-ssh--line-cleanup (str)
  "Return STR as a list of strings."
  (let* ((lst-contents (thread-last str
                                    (s-split "\n")
                                    (-remove 's-blank-str-p)))
         (first-line   (car lst-contents))
         (trim-amount  (when (string-match (rx bol (group (* space))) first-line)
                         (length (match-string 1 first-line)))))
    (mapcar (lambda (line) (substring line trim-amount)) lst-contents)))

(ert-deftest ha-ssh--line-cleanup-test ()
  (should (equal (ha-ssh--line-cleanup "bob") '("bob")))
  (should (equal (ha-ssh--line-cleanup "  bob") '("bob")))
  (should (equal (ha-ssh--line-cleanup "bob\nfoo") '("bob" "foo")))
  (should (equal (ha-ssh--line-cleanup "  bob\n  foo") '("bob" "foo")))
  (should (equal (ha-ssh--line-cleanup "  bob\n    foo") '("bob" "  foo"))))

Editing Remote Files

TRAMP, when it works, is amazing that we can give it a reference to a remote directory, and have find-file magically autocomplete.

(defun ha-ssh-find-file (hostname)
  "Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=."
  (interactive (list (ha-ssh-choose-host)))
  (let ((tramp-ssh-ref (format "/ssh:%s:" hostname))
        (other-window (when (equal current-prefix-arg '(4)) t)))
    (ha-ssh--find-file tramp-ssh-ref other-window)))

(defun ha-ssh--find-file (tramp-ssh-ref &optional other-window)
  "Calls =find-file= after internally completing a file reference based on TRAMP-SSH-REF."
  (let ((tramp-file (read-file-name "Find file: " tramp-ssh-ref)))
    (if other-window
        (find-file-other-window tramp-file)
      (find-file tramp-file))))

We can even edit it as root:

(defun ha-ssh-find-root (hostname)
  "Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=."
  (interactive (list (ha-ssh-choose-host)))
  (let ((tramp-ssh-ref (format "/ssh:%s|sudo:%s:" hostname hostname))
        (other-window (when (equal current-prefix-arg '(4)) t)))
    (ha-ssh--find-file tramp-ssh-ref other-window)))

OpenStack Interface

Instead of making sure I have a list of remote systems already in the favorite hosts cache, I can pre-populate it with a call to OpenStack (my current VM system I’m using). These calls to the openstack CLI assume that the environment is already filled with the credentials. Hey, it is my local laptop …

We’ll give openstack CLI a --format json option to make it easier for parsing:

(use-package json)

Need a variable to hold all our interesting hosts. Notice I use the word overcloud, but this is a name I’ve used for years to refer to my virtual machines that I can get a listing of, and not get other VMs that I don’t own.

(defvar ha-ssh-overcloud-cache-data nil
  "A vector of associated lists containing the servers in an Overcloud.")

If our cache data is empty, we could automatically retrieve this information, but only on the first time we attempt to connect. To do this, we’ll advice the ha-ssh-choose-host function defined earlier:

(defun ha-ssh-overcloud-query-for-hosts ()
  "If the overcloud cache hasn't be populated, ask the user if we want to run the command."
  (when (not ha-ssh-overcloud-cache-data)
    (when (y-or-n-p "Cache of Overcloud hosts aren't populated. Retrieve hosts?")
      (call-interactively 'ha-ssh-overcloud-cache-populate))))

(advice-add 'ha-ssh-choose-host :before 'ha-ssh-overcloud-query-for-hosts)

We’ll do the work of getting the server list with this function:

(defun ha-ssh-overcloud-cache-populate (cluster)
  "Given an `os-cloud' entry, stores all available hostnames.
Calls `ha-ssh-add-favorite-host' for each host found."
  (interactive (list (completing-read "Cluster: " '(devprod1 devprod501 devprod502))))
  (message "Calling the `openstack' command...this will take a while. Grab a coffee, eh?")
  (let* ((command (format "openstack --os-cloud %s server list --no-name-lookup --insecure -f json" cluster))
         (json-data (thread-last command
                                 (shell-command-to-string)
                                 (json-read-from-string))))
    (dolist (entry (seq--into-list json-data))
      (ha-ssh-add-favorite-host (alist-get 'Name entry)
                                (thread-last entry
                                             (alist-get 'Networks)
                                             (alist-get 'cedev13)
                                             (seq-first))))
    (message "Call to `openstack' complete. Found %d hosts." (length json-data))))

In case I change my virtual machines, I can repopulate that cache:

(defun ha-ssh-overcloud-cache-repopulate ()
  "Repopulate the cache based on redeployment of my overcloud."
  (interactive)
  (setq ha-ssh-overcloud-cache-data nil)
  (call-interactively 'ha-ssh-overcloud-cache-populate))

The primary interface:

(defun ha-ssh-overcloud (hostname)
  "Log into an overcloud host given by HOSTNAME. Works better if
you have previously run =ssh-copy-id= on the host. Remember, to
make it behave like a real terminal (instead of a window in
Emacs), hit =C-c C-k=."
  (interactive (list (ha-ssh-choose-host)))
  (when (not (string-match-p "\." hostname))
    (setq hostname (format "%s.%s" hostname (getenv "OS_PROJECT_NAME"))))

  (let ((window-label (or (-some->> ha-ssh-favorite-hostnames
                                    (rassoc hostname)
                                    car)
                          hostname)))
    (ha-ssh hostname window-label)
    (sit-for 1)
    (ha-ssh-send "sudo -i")
    (ha-ssh-send (format "export PS1='\\[\\e[34m\\]%s\\[\e[m\\] \\[\\e[33m\\]\\$\\[\\e[m\\] '"
                         window-label))
    (ha-ssh-send "clear")))

Keybindings

This file, so far, as been good-enough for a Vanilla Emacs installation, but to hook into Doom’s leader for some sequence binding, this code isn’t:

(ha-leader
  "a s"  '(:ignore t :which-key "ssh")
  "a s o" '("overcloud"     . ha-ssh-overcloud)
  "a s l" '("local shell"   . ha-shell)
  "a s s" '("remote shell"  . ha-ssh)
  "a s p" '("project shell" . (lambda () (interactive) (ha-shell (project-root (project-current)))))
  "a s q" '("quit shell"    . ha-ssh-exit)
  "a s f" '("find-file"     . ha-ssh-find-file)
  "a s r" '("find-root"     . ha-ssh-find-root)
  "a s b" '("send line"     . ha-ssh-send-line)

  "p t"   '("project vterm" . (lambda () (interactive) (ha-shell (project-root (project-current))))))