#+TITLE: Remote Access to Systems
#+AUTHOR: Howard X. Abrams
#+DATE: 2020-09-25
A literate configuration for accessing remote systems.
#+begin_src emacs-lisp :exports none
;;; ha-remoting --- Accessing remote systems. -*- lexical-binding: t; -*-
;;
;; © 2020-2022 Howard X. Abrams
;; Licensed under a Creative Commons Attribution 4.0 International License.
;; See http://creativecommons.org/licenses/by/4.0/
;;
;; Author: Howard X. Abrams <http://gitlab.com/howardabrams>
;; Maintainer: Howard X. Abrams
;; Created: September 25, 2020
;;
;; This file is not part of GNU Emacs.
;;
;; *NB:* Do not edit this file. Instead, edit the original literate file at:
;; ~/other/hamacs/ha-remoting.org
;; And tangle the file to recreate this one.
;;
;;; Code:
#+end_src
* Local Terminals
The following section configures my Terminal experience, both inside and outside Emacs.
** Eshell
I used to use [[http://www.howardism.org/Technical/Emacs/eshell.html][Eshell all the time]], but now I've migrated most of /work/ directly into Emacs (rewriting all those shell scripts a Emacs Lisp code). A shell is pretty good for my brain at organizing files (old habits, maybe).
First, my /aliases/ follow me around, and the following creates the alias file, =~/.emacs.d/eshell/alias=:
#+begin_src shell :tangle (identity eshell-aliases-file) :mkdirp yes
alias ll ls -l $*
alias clear recenter 0
alias emacs 'find-file $1'
alias e 'find-file $1'
alias ee 'find-file-other-window $1'
alias grep 'rg -n -H --no-heading -e "$*"'
alias find 'echo Please use fd instead.' # :-)
#+end_src
Since that file already exists (probably), the following command may not be necessary:
#+begin_src emacs-lisp
(use-package eshell
:custom
(eshell-kill-on-exit t)
(eshell-destroy-buffer-when-process-dies t)
:hook
(eshell-exit . (lambda () (delete-window))))
#+end_src
I usually want a new window running Eshell, that is smaller than the current buffer:
#+begin_src emacs-lisp
(defun eshell-there (parent)
"Open an eshell session in a PARENT directory
in a smaller window named after this directory."
(let ((name (thread-first parent
(split-string "/" t)
(last)
(car)))
(height (/ (window-total-height) 3)))
(split-window-vertically (- height))
(eshell "new")
(rename-buffer (concat "*eshell: " name "*"))
(insert (concat "ls"))
(eshell-send-input)))
#+end_src
We either want to start the shell in the same parent as the current buffer, or at the root of the project:
#+begin_src emacs-lisp
(defun eshell-here ()
"Opens up a new shell in the directory associated with the
current buffer's file. Rename the eshell buffer name to match
that directory to make multiple eshell windows easier."
(interactive)
(eshell-there (if (buffer-file-name)
(file-name-directory (buffer-file-name))
default-directory)))
(defun eshell-project ()
"Open a new shell in the project root directory, in a smaller window."
(interactive)
(eshell-there (projectile-project-root)))
#+end_src
Add my org-specific predicates, see this [[http://www.howardism.org/Technical/Emacs/eshell-fun.html][this essay]] for the details:
#+begin_src emacs-lisp
(defun eshell-org-file-tags ()
"Helps the eshell parse the text the point is currently on,
looking for parameters surrounded in single quotes. Returns a
function that takes a FILE and returns nil if the file given to
it doesn't contain the org-mode #+FILETAGS: entry specified."
;; Step 1. Parse the eshell buffer for our tag between quotes
;; Make sure to move point to the end of the match:
(if (looking-at "'\\([^)']+\\)'")
(let* ((tag (match-string 1))
(reg (rx bol "#+FILETAGS: "
(zero-or-more any)
word-start
(literal tag)
word-end
(zero-or-more any)
eol)))
(goto-char (match-end 0))
;; Step 2. Return the predicate function:
;; Careful when accessing the `reg' variable.
`(lambda (file)
(with-temp-buffer
(insert-file-contents file)
(re-search-forward ,reg nil t 1))))
(error "The `T' predicate takes an org-mode tag value in single quotes.")))
(defvar eshell-predicate-alist nil
"A list of predicates than can be applied to a globbing pattern.")
(add-to-list 'eshell-predicate-alist '(?T . (eshell-org-file-tags)))
#+end_src
* Remote Editing with Tramp
[[https://www.emacswiki.org/emacs/TrampMode][Tramp]] allows almost all Emacs features to execute on a remote system.
#+begin_src emacs-lisp
(use-package tramp
:straight (:type built-in)
:config
;; Use remote PATH on tramp (handy for eshell).
(add-to-list 'tramp-remote-path 'tramp-own-remote-path)
;; Make sure version control system doesn't slow tramp:
(setq vc-ignore-dir-regexp
(format "%s\\|%s" vc-ignore-dir-regexp tramp-file-name-regexp)))
#+end_src
Will Schenk has [[https://willschenk.com/articles/2020/tramp_tricks/][a simple extension]] to allow editing of files /inside/ a Docker container:
#+begin_src emacs-lisp
(use-package tramp
:straight (:type built-in)
:config
(push '("docker" . ((tramp-login-program "docker")
(tramp-login-args (("exec" "-it") ("%h") ("/bin/sh")))
(tramp-remote-shell "/bin/sh")
(tramp-remote-shell-args ("-i") ("-c"))))
tramp-methods)
(defadvice tramp-completion-handle-file-name-all-completions
(around dotemacs-completion-docker activate)
"(tramp-completion-handle-file-name-all-completions \"\" \"/docker:\" returns
a list of active Docker container names, followed by colons."
(if (equal (ad-get-arg 1) "/docker:")
(let* ((command "docker ps --format '{{.Names}}:'")
(dockernames-raw (shell-command-to-string command))
(dockernames (split-string dockernames-raw "\n")))
(setq ad-return-value dockernames))
ad-do-it)))
#+end_src
Keep in mind you need to /name/ your Docker session, with the =—name= option. I actually do more docker work on remote systems (as Docker seems to make my fans levitate my laptop over the desk). Granted, the =URL= is a bit lengthy, for instance:
#+begin_example
/ssh:kolla-compute1.cedev13.d501.eng.pdx.wd|sudo:kolla-compute1.cedev13.d501.eng.pdx.wd|docker:kolla_toolbox:/
#+end_example
Which means, I need to put it as a link in an org file.
*Note:* That we need to have Tramp SSH option comes from my personal [[file:~/.ssh/config][.ssh/config]] file instead of its internal cache:
#+begin_src emacs-lisp
(use-package tramp-sh
:after tramp
:straight (:type built-in)
:custom (tramp-use-ssh-controlmaster-options nil))
#+end_src
* Remote Terminals
Sure =iTerm= is nice for connecting and running commands on remote systems, however, it lacks a command line option that allows you to select and manipulate the displayed text without a mouse. This is where Emacs can shine.
*Feature One:*
When calling the =ha-ssh= function, it opens a =vterm= window which, unlike other terminal emulators in Emacs, merges both Emacs and Terminal behaviors. Essentially, it just works. It =vterm= isn't installed, it falls back to =term=.
Preload a list of favorite/special hostnames with multiple calls to:
#+begin_src emacs-lisp :tangle no
(ha-ssh-add-favorite-host "Devbox 42" "10.0.1.42")
#+end_src
Then calling =ha-ssh= function, a list of hostnames is available to quickly jump on a system (with the possibility of fuzzy matching if you have Helm or Ivy installed).
This also has the ability to call OpenStack to gather the hostnames of dynamic systems (what I call "an Overcloud"), which is appended to the list of favorite hostnames. The call to OpenStack only needs to be called once, since the hosts are then cached, see =ha-ssh-overcloud-query-for-hosts=.
*Feature Two:*
Use the /favorite host/ list to quickly edit a file on a remote system using Tramp, by calling either =ha-ssh-find-file= and =ha-ssh-find-root=.
*Feature Three:*
Working with remote shell connections programmatically, for instance:
#+begin_src emacs-lisp :tangle no
(let ((win-name "some-host"))
(ha-ssh "some-host.in.some.place" win-name)
(ha-ssh-send "source ~/.bash_profile" win-name)
(ha-ssh-send "clear" win-name))
;; ...
(ha-ssh-exit win-name)
#+end_src
Actually the =win-name= in this case is optional, as it will use a good default.
** VTerm
I'm not giving up on Eshell, but I am playing around with [[https://github.com/akermu/emacs-libvterm][vterm]], and it is pretty good, but I use it primarily as a more reliable approach for remote terminal sessions.
VTerm has an issue (at least for me) with ~M-Backspace~ not deleting the previous word, and yeah, I want to make sure that both keystrokes do the same thing.
#+begin_src emacs-lisp
(use-package vterm
:init
(setq vterm-shell "/usr/local/bin/fish")
;; Granted, I seldom pop out to the shell except during code demonstrations,
;; but I like how C-p/C-n jumps up to each prompt entry using this setting
;; that works with my prompt:
(setq vterm-use-vterm-prompt-detection-method nil
term-prompt-regexp "^.* $ ")
:config
(dolist (k '("<C-backspace>" "<M-backspace>"))
(define-key vterm-mode-map (kbd k)
(lambda () (interactive) (vterm-send-key (kbd "C-w")))))
(advice-add 'vterm-copy-mode :after 'evil-normal-state))
#+end_src
The advantage of running terminals in Emacs is the ability to copy text without a mouse. For that, hit ~C-c C-t~ to enter a special copy-mode. If I go into this mode, I might as well also go into normal mode to move the cursor.
*Note:* To exit the copy-mode (and copy the selected text to the clipboard), hit ~Return~.
Hrm. Seems that I might want a function to copy the output of the last command to a register, or even an org-capture...
** Variables
Let's begin by defining some variables used for communication between the functions.
#+begin_src emacs-lisp
(defvar ha-latest-ssh-window-name nil
"The window-name of the latest ssh session. Most commands default to the last session.")
(defvar ha-ssh-host-history '() "List of hostnames we've previously connected.")
(defvar ha-ssh-favorite-hostnames '()
"A list of tuples (associate list) containing a hostname and its IP address.
See =ha-ssh-add-favorite-host= for easily adding to this list.")
#+end_src
Also, let's make it easy for me to change my default shell:
#+begin_src emacs-lisp
(defvar ha-ssh-shell (shell-command-to-string "type -p fish")
"The executable to the shell I want to use locally.")
#+end_src
** Interactive Interface to Remote Systems
The function, =ha-ssh= pops up a list of /favorite hosts/ and then uses the =vterm= functions to automatically SSH into the chosen host:
#+begin_src emacs-lisp
(defun ha-ssh (hostname &optional window-name)
"Start a SSH session to a given HOSTNAME (with an optionally specified WINDOW-NAME).
If called interactively, it presents the user with a list
returned by =ha-ssh-choose-host=."
(interactive (list (ha-ssh-choose-host)))
(unless window-name
(setq window-name (format "ssh: %s" hostname)))
(setq ha-latest-ssh-window-name (format "*%s*" window-name))
;; I really like this =vterm= interface, so if I've got it loaded, let's use it:
(if (not (fboundp 'vterm))
;; Should we assume the =ssh= we want is on the PATH that started Emacs?
(make-term window-name "ssh" nil hostname)
(vterm ha-latest-ssh-window-name)
(vterm-send-string (format "ssh %s" hostname))
(vterm-send-return))
(pop-to-buffer ha-latest-ssh-window-name))
#+end_src
Of course, we need a function that =interactive= can call to get that list, and my thought is to call =helm= if it is available, otherwise, assume that ido/ivy will take over the =completing-read= function:
#+begin_src emacs-lisp
(defun ha-ssh-choose-host ()
"Prompts the user for a host, and if it is in the cache, return
its IP address, otherwise, return the input given.
This is used in calls to =interactive= to select a host."
(let ((hostname
;; We call Helm directly if installed, only so that we can get better
;; labels in the window, otherwise, the =completing-read= call would be fine.
(if (fboundp 'helm-comp-read)
(helm-comp-read "Hostname: " ha-ssh-favorite-hostnames
:name "Hosts"
:fuzzy t :history ha-ssh-host-history)
(completing-read "Hostname: " ha-ssh-favorite-hostnames nil 'confirm nil 'ha-ssh-host-history))))
(alist-get hostname ha-ssh-favorite-hostnames hostname nil 'equal)))
#+end_src
Simply calling =vterm= fails to load my full environment, so this allows me to start the terminal in a particular directory (defaulting to the root of the current project):
#+begin_src emacs-lisp
(defun ha-shell (&optional directory)
"Creates and tidies up a =vterm= terminal shell in side window."
(interactive (list (read-directory-name "Starting Directory: " (projectile-project-root))))
(let* ((win-name "Terminal")
(buf-name (format "*%s*" win-name))
(default-directory (or directory default-directory)))
(setq ha-latest-ssh-window-name buf-name)
(if (not (fboundp 'vterm))
(make-term win-name ha-ssh-shell)
(vterm buf-name)
;; (ha-ssh-send "source ~/.bash_profile" buf-name)
;; (ha-ssh-send "clear" buf-name)
)))
#+end_src
Before we leave this section, I realize that I would like a way to /add/ to my list of hosts:
#+begin_src emacs-lisp
(defun ha-ssh-add-favorite-host (hostname ip-address)
"Add a favorite host to your list for easy pickin's."
(interactive "sHostname: \nsIP Address: ")
(add-to-list 'ha-ssh-favorite-hostnames (cons hostname ip-address)))
#+end_src
** Programmatic Interface
The previous functions (as well as my own end of sprint demonstrations) often need to issue some commands to a running terminal session, which is a simple wrapper around a /send text/ and /send return/ sequence:
#+begin_src emacs-lisp
(defun ha-ssh-send (phrase &optional window-name)
"Send command PHRASE to the currently running SSH instance.
If you want to refer to another session, specify the correct WINDOW-NAME.
This is really useful for scripts and demonstrations."
(unless window-name
(setq window-name ha-latest-ssh-window-name))
(pop-to-buffer window-name)
(if (fboundp 'vterm)
(progn
(vterm-send-string phrase)
(vterm-send-return))
(progn
(term-send-raw-string phrase)
(term-send-input))))
#+end_src
On the rare occasion that I write a shell script, or at least, need to execute some one-line shell commands from some document, I have a function that combines a /read line from buffer/ and then send it to the currently running terminal:
#+begin_src emacs-lisp
(defun ha-ssh-send-line ()
"Copy the contents of the current line in the current buffer,
and call =ha-ssh-send= with it. After sending the contents, it
returns to the current line."
(interactive)
;; The function =save-excursion= doesn't seem to work...
(let* ((buf (current-buffer))
(cmd-line (buffer-substring-no-properties
(line-beginning-position) (line-end-position)))
(trim-cmd (s-trim cmd-line)))
(ha-ssh-send trim-cmd)
;; (sit-for 0.25)
(pop-to-buffer buf)))
#+end_src
Let's have a quick way to bugger out of the terminal:
#+begin_src emacs-lisp
(defun ha-ssh-exit (&optional window-name)
"End the SSH session specified by WINDOW-NAME (or if not, the latest session)."
(interactive)
(unless (string-match-p "v?term" (buffer-name))
(unless window-name
(setq window-name ha-latest-ssh-window-name))
(pop-to-buffer window-name))
(ignore-errors
(term-send-eof))
(kill-buffer window-name)
(delete-window))
#+end_src
** Editing Remote Files
TRAMP, when it works, is amazing that we can give it a reference to a remote directory, and have =find-file= magically autocomplete.
#+begin_src emacs-lisp
(defun ha-ssh-find-file (hostname)
"Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=."
(interactive (list (ha-ssh-choose-host)))
(let ((tramp-ssh-ref (format "/ssh:%s:" hostname))
(other-window (when (equal current-prefix-arg '(4)) t)))
(ha-ssh--find-file tramp-ssh-ref other-window)))
(defun ha-ssh--find-file (tramp-ssh-ref &optional other-window)
"Calls =find-file= after internally completing a file reference based on TRAMP-SSH-REF."
(let ((tramp-file (read-file-name "Find file: " tramp-ssh-ref)))
(if other-window
(find-file-other-window tramp-file)
(find-file tramp-file))))
#+end_src
We can even edit it as root:
#+begin_src emacs-lisp
(defun ha-ssh-find-root (hostname)
"Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=."
(interactive (list (ha-ssh-choose-host)))
(let ((tramp-ssh-ref (format "/ssh:%s|sudo:%s:" hostname hostname))
(other-window (when (equal current-prefix-arg '(4)) t)))
(ha-ssh--find-file tramp-ssh-ref other-window)))
#+end_src
** OpenStack Interface
Instead of making sure I have a list of remote systems already in the favorite hosts cache, I can pre-populate it with a call to OpenStack (my current VM system I'm using). These calls to the =openstack= CLI assume that the environment is already filled with the credentials. Hey, it is my local laptop ...
We'll give =openstack= CLI a =--format json= option to make it easier for parsing:
#+begin_src emacs-lisp
(use-package json)
#+end_src
Need a variable to hold all our interesting hosts. Notice I use the word /overcloud/, but this is a name I've used for years to refer to /my virtual machines/ that I can get a listing of, and not get other VMs that I don't own.
#+begin_src emacs-lisp
(defvar ha-ssh-overcloud-cache-data nil
"A vector of associated lists containing the servers in an Overcloud.")
#+end_src
If our cache data is empty, we could automatically retrieve this information, but only on the first time we attempt to connect. To do this, we'll =advice= the =ha-ssh-choose-host= function defined earlier:
#+begin_src emacs-lisp
(defun ha-ssh-overcloud-query-for-hosts ()
"If the overcloud cache hasn't be populated, ask the user if we want to run the command."
(when (not ha-ssh-overcloud-cache-data)
(when (y-or-n-p "Cache of Overcloud hosts aren't populated. Retrieve hosts?")
(call-interactively 'ha-ssh-overcloud-cache-populate))))
(advice-add 'ha-ssh-choose-host :before 'ha-ssh-overcloud-query-for-hosts)
#+end_src
We'll do the work of getting the /server list/ with this function:
#+begin_src emacs-lisp
(defun ha-ssh-overcloud-cache-populate (cluster)
"Given an `os-cloud' entry, stores all available hostnames.
Calls `ha-ssh-add-favorite-host' for each host found."
(interactive (list (completing-read "Cluster: " '(devprod1 devprod501 devprod502))))
(message "Calling the `openstack' command...this will take a while. Grab a coffee, eh?")
(let* ((command (format "openstack --os-cloud %s server list --no-name-lookup --insecure -f json" cluster))
(json-data (thread-last command
(shell-command-to-string)
(json-read-from-string))))
(dolist (entry (seq--into-list json-data))
(ha-ssh-add-favorite-host (alist-get 'Name entry)
(thread-last entry
(alist-get 'Networks)
(alist-get 'cedev13)
(seq-first))))
(message "Call to `openstack' complete. Found %d hosts." (length json-data))))
#+end_src
In case I change my virtual machines, I can repopulate that cache:
#+begin_src emacs-lisp
(defun ha-ssh-overcloud-cache-repopulate ()
"Repopulate the cache based on redeployment of my overcloud."
(interactive)
(setq ha-ssh-overcloud-cache-data nil)
(call-interactively 'ha-ssh-overcloud-cache-populate))
#+end_src
The primary interface:
#+begin_src emacs-lisp
(defun ha-ssh-overcloud (hostname)
"Log into an overcloud host given by HOSTNAME. Works better if
you have previously run =ssh-copy-id= on the host. Remember, to
make it behave like a real terminal (instead of a window in
Emacs), hit =C-c C-k=."
(interactive (list (ha-ssh-choose-host)))
(when (not (string-match-p "\." hostname))
(setq hostname (format "%s.%s" hostname (getenv "OS_PROJECT_NAME"))))
(let ((window-label (or (-some->> ha-ssh-favorite-hostnames
(rassoc hostname)
car)
hostname)))
(ha-ssh hostname window-label)
(sit-for 1)
(ha-ssh-send "sudo -i")
(ha-ssh-send (format "export PS1='\\[\\e[34m\\]%s\\[\e[m\\] \\[\\e[33m\\]\\$\\[\\e[m\\] '"
window-label))
(ha-ssh-send "clear")))
#+end_src
* Keybindings
This file, so far, as been good-enough for a Vanilla Emacs installation, but to hook into Doom's leader for some sequence binding, this code isn't:
#+begin_src emacs-lisp
(ha-leader
"a e" '("eshell" . eshell-here)
"a E" '("top eshell" . eshell-project)
"a s" '(:ignore t :which-key "ssh")
"a s v" '("vterm" . vterm)
"a s o" '("overcloud" . ha-ssh-overcloud)
"a s l" '("local shell" . ha-shell)
"a s s" '("remote shell" . ha-ssh)
"a s q" '("quit shell" . ha-ssh-exit)
"a s f" '("find-file" . ha-ssh-find-file)
"a s r" '("find-root" . ha-ssh-find-root))
#+end_src
* Technical Artifacts :noexport:
Provide a name so we can =require= the file:
#+begin_src emacs-lisp :exports none
(provide 'ha-remoting)
;;; ha-remoting.el ends here
#+end_src
Before you can build this on a new system, make sure that you put the cursor over any of these properties, and hit: ~C-c C-c~
#+DESCRIPTION: A literate configuration for accessing remote systems.
#+PROPERTY: header-args:sh :tangle no
#+PROPERTY: header-args:emacs-lisp :tangle yes
#+PROPERTY: header-args :results none :eval no-export :comments no mkdirp yes
#+OPTIONS: num:nil toc:nil todo:nil tasks:nil tags:nil date:nil
#+OPTIONS: skip:nil author:nil email:nil creator:nil timestamp:nil
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js