#+TITLE: Remote Access to Systems #+AUTHOR: Howard X. Abrams #+DATE: 2020-09-25 #+FILETAGS: :emacs: A literate configuration for accessing remote systems. #+BEGIN_SRC emacs-lisp :exports none ;;; ha-remoting --- Accessing remote systems. -*- lexical-binding: t; -*- ;; ;; © 2020-2022 Howard X. Abrams ;; This work is licensed under a Creative Commons Attribution 4.0 International License. ;; See http://creativecommons.org/licenses/by/4.0/ ;; ;; Author: Howard X. Abrams ;; Maintainer: Howard X. Abrams ;; Created: September 25, 2020 ;; ;; This file is not part of GNU Emacs. ;; ;; *NB:* Do not edit this file. Instead, edit the original literate file at: ;; ~/other/hamacs/ha-remoting.org ;; And tangle the file to recreate this one. ;; ;;; Code: #+END_SRC * Local Terminals The following section configures my Terminal experience, both inside and outside Emacs. ** Eshell I used to use [[http://www.howardism.org/Technical/Emacs/eshell.html][Eshell all the time]], but now I've migrated most of /work/ directly into Emacs (rewriting all those shell scripts a Emacs Lisp code). However, a shell is pretty good for my brain at organizing files (old habits, maybe). First, my /aliases/ follow me around, and the following creates the alias file, =~/.emacs.d/eshell/alias=: #+BEGIN_SRC shell :tangle (identity eshell-aliases-file) :mkdirp yes alias ll ls -l $* alias clear recenter 0 alias emacs 'find-file $1' alias e 'find-file $1' alias ee 'find-file-other-window $1' alias grep 'rg -n -H --no-heading -e "$*"' alias find 'echo Please use fd instead.' # :-) #+END_SRC Since that file already exists (probably), the following command may not be necessary: #+BEGIN_SRC emacs-lisp (use-package eshell :custom (eshell-kill-on-exit t) (eshell-destroy-buffer-when-process-dies t) :hook (eshell-exit . (lambda () (delete-window)))) #+END_SRC I usually want a new window running Eshell, that is smaller than the current buffer: #+BEGIN_SRC emacs-lisp (defun eshell-there (parent) "Open an eshell session in a PARENT directory in a smaller window named after this directory." (let ((name (thread-first parent (split-string "/" t) (last) (car))) (height (/ (window-total-height) 3))) (split-window-vertically (- height)) (eshell "new") (rename-buffer (concat "*eshell: " name "*")) (insert (concat "ls")) (eshell-send-input))) #+END_SRC We either want to start the shell in the same parent as the current buffer, or at the root of the project: #+BEGIN_SRC emacs-lisp (defun eshell-here () "Opens up a new shell in the directory associated with the current buffer's file. Rename the eshell buffer name to match that directory to make multiple eshell windows easier." (interactive) (eshell-there (if (buffer-file-name) (file-name-directory (buffer-file-name)) default-directory))) (defun eshell-project () "Open a new shell in the project root directory, in a smaller window." (interactive) (eshell-there (projectile-project-root))) #+END_SRC Add my org-specific predicates, see this [[http://www.howardism.org/Technical/Emacs/eshell-fun.html][this essay]] for the details: #+BEGIN_SRC emacs-lisp (defun eshell-org-file-tags () "Helps the eshell parse the text the point is currently on, looking for parameters surrounded in single quotes. Returns a function that takes a FILE and returns nil if the file given to it doesn't contain the org-mode #+FILETAGS: entry specified." ;; Step 1. Parse the eshell buffer for our tag between quotes ;; Make sure to move point to the end of the match: (if (looking-at "'\\([^)']+\\)'") (let* ((tag (match-string 1)) (reg (rx bol "#+FILETAGS: " (zero-or-more any) word-start (literal tag) word-end (zero-or-more any) eol))) (goto-char (match-end 0)) ;; Step 2. Return the predicate function: ;; Careful when accessing the `reg' variable. `(lambda (file) (with-temp-buffer (insert-file-contents file) (re-search-forward ,reg nil t 1)))) (error "The `T' predicate takes an org-mode tag value in single quotes."))) (defvar eshell-predicate-alist nil "A list of predicates than can be applied to a globbing pattern.") (add-to-list 'eshell-predicate-alist '(?T . (eshell-org-file-tags))) #+END_SRC * Remote Terminals Sure =iTerm= is nice for connecting and running commands on remote systems, however, it lacks a command line option that allows you to select and manipulate the displayed text without a mouse. This is where Emacs can shine. *Feature One:* When calling the =ha-ssh= function, it opens a =vterm= window which, unlike other terminal emulators in Emacs, merges both Emacs and Terminal behaviors. Essentially, it just works. It =vterm= isn't installed, it falls back to =term=. Preload a list of favorite/special hostnames with multiple calls to: #+BEGIN_SRC emacs-lisp :tangle no (ha-ssh-add-favorite-host "Devbox 42" "10.0.1.42") #+END_SRC Then calling =ha-ssh= function, a list of hostnames is available to quickly jump on a system (with the possibility of fuzzy matching if you have Helm or Ivy installed). This also has the ability to call OpenStack to gather the hostnames of dynamic systems (what I call "an Overcloud"), which is appended to the list of favorite hostnames. The call to OpenStack only needs to be called once, since the hosts are then cached, see =ha-ssh-overcloud-query-for-hosts=. *Feature Two:* Use the /favorite host/ list to quickly edit a file on a remote system using Tramp, by calling either =ha-ssh-find-file= and =ha-ssh-find-root=. *Feature Three:* Working with remote shell connections programmatically, for instance: #+BEGIN_SRC emacs-lisp :tangle no (let ((win-name "some-host")) (ha-ssh "some-host.in.some.place" win-name) (ha-ssh-send "source ~/.bash_profile" win-name) (ha-ssh-send "clear" win-name)) ;; ... (ha-ssh-exit win-name) #+END_SRC Actually the =win-name= in this case is optional, as it will use a good default. ** VTerm I'm not giving up on Eshell, but I am playing around with [[https://github.com/akermu/emacs-libvterm][vterm]], and it is pretty good, but I use it primarily as a more reliable approach for remote terminal sessions. VTerm has an issue (at least for me) with ~M-Backspace~ not deleting the previous word, and yeah, I want to make sure that both keystrokes do the same thing. #+BEGIN_SRC emacs-lisp (use-package vterm :init (setq vterm-shell "/usr/local/bin/fish") ;; Granted, I seldom pop out to the shell except during code demonstrations, ;; but I like how C-p/C-n jumps up to each prompt entry using this setting ;; that works with my prompt: (setq vterm-use-vterm-prompt-detection-method nil term-prompt-regexp "^.* $ ") :config (dolist (k '("" "")) (define-key vterm-mode-map (kbd k) (lambda () (interactive) (vterm-send-key (kbd "C-w"))))) (advice-add 'vterm-copy-mode :after 'evil-normal-state)) #+END_SRC The advantage of running terminals in Emacs is the ability to copy text without a mouse. For that, hit ~C-c C-t~ to enter a special copy-mode. If I go into this mode, I might as well also go into normal mode to move the cursor. *Note:* To exit the copy-mode (and copy the selected text to the clipboard), hit ~Return~. Hrm. Seems that I might want a function to copy the output of the last command to a register, or even an org-capture... ** Variables Let's begin by defining some variables used for communication between the functions. #+BEGIN_SRC emacs-lisp (defvar ha-latest-ssh-window-name nil "The window-name of the latest ssh session. Most commands default to the last session.") (defvar ha-ssh-host-history '() "List of hostnames we've previously connected.") (defvar ha-ssh-favorite-hostnames '() "A list of tuples (associate list) containing a hostname and its IP address. See =ha-ssh-add-favorite-host= for easily adding to this list.") #+END_SRC Also, let's make it easy for me to change my default shell: #+BEGIN_SRC emacs-lisp (defvar ha-ssh-shell (shell-command-to-string "type -p fish") "The executable to the shell I want to use locally.") #+END_SRC ** Interactive Interface to Remote Systems The function, =ha-ssh= pops up a list of /favorite hosts/ and then uses the =vterm= functions to automatically SSH into the chosen host: #+BEGIN_SRC emacs-lisp (defun ha-ssh (hostname &optional window-name) "Start a SSH session to a given HOSTNAME (with an optionally specified WINDOW-NAME). If called interactively, it presents the user with a list returned by =ha-ssh-choose-host=." (interactive (list (ha-ssh-choose-host))) (unless window-name (setq window-name (format "ssh: %s" hostname))) (setq ha-latest-ssh-window-name (format "*%s*" window-name)) ;; I really like this =vterm= interface, so if I've got it loaded, let's use it: (if (not (fboundp 'vterm)) ;; Should we assume the =ssh= we want is on the PATH that started Emacs? (make-term window-name "ssh" nil hostname) (vterm ha-latest-ssh-window-name) (vterm-send-string (format "ssh %s" hostname)) (vterm-send-return)) (pop-to-buffer ha-latest-ssh-window-name)) #+END_SRC Of course, we need a function that =interactive= can call to get that list, and my thought is to call =helm= if it is available, otherwise, assume that ido/ivy will take over the =completing-read= function: #+BEGIN_SRC emacs-lisp (defun ha-ssh-choose-host () "Prompts the user for a host, and if it is in the cache, return its IP address, otherwise, return the input given. This is used in calls to =interactive= to select a host." (let ((hostname ;; We call Helm directly if installed, only so that we can get better ;; labels in the window, otherwise, the =completing-read= call would be fine. (if (fboundp 'helm-comp-read) (helm-comp-read "Hostname: " ha-ssh-favorite-hostnames :name "Hosts" :fuzzy t :history ha-ssh-host-history) (completing-read "Hostname: " ha-ssh-favorite-hostnames nil 'confirm nil 'ha-ssh-host-history)))) (alist-get hostname ha-ssh-favorite-hostnames hostname nil 'equal))) #+END_SRC Simply calling =vterm= fails to load my full environment, so this allows me to start the terminal in a particular directory (defaulting to the root of the current project): #+BEGIN_SRC emacs-lisp (defun ha-shell (&optional directory) "Creates and tidies up a =vterm= terminal shell in side window." (interactive (list (counsel-read-directory-name "Starting Directory:" (projectile-project-root)))) (let* ((win-name "Terminal") (buf-name (format "*%s*" win-name)) (default-directory (or directory default-directory))) (setq ha-latest-ssh-window-name buf-name) (if (not (fboundp 'vterm)) (make-term win-name ha-ssh-shell) (vterm buf-name) ;; (ha-ssh-send "source ~/.bash_profile" buf-name) ;; (ha-ssh-send "clear" buf-name) ))) #+END_SRC Before we leave this section, I realize that I would like a way to /add/ to my list of hosts: #+BEGIN_SRC emacs-lisp (defun ha-ssh-add-favorite-host (hostname ip-address) "Add a favorite host to your list for easy pickin's." (interactive "sHostname: \nsIP Address: ") (add-to-list 'ha-ssh-favorite-hostnames (cons hostname ip-address))) #+END_SRC ** Programmatic Interface The previous functions (as well as my own end of sprint demonstrations) often need to issue some commands to a running terminal session, which is a simple wrapper around a /send text/ and /send return/ sequence: #+BEGIN_SRC emacs-lisp (defun ha-ssh-send (phrase &optional window-name) "Send command PHRASE to the currently running SSH instance. If you want to refer to another session, specify the correct WINDOW-NAME. This is really useful for scripts and demonstrations." (unless window-name (setq window-name ha-latest-ssh-window-name)) (pop-to-buffer window-name) (if (fboundp 'vterm) (progn (vterm-send-string phrase) (vterm-send-return)) (progn (term-send-raw-string phrase) (term-send-input)))) #+END_SRC On the rare occasion that I write a shell script, or at least, need to execute some one-line shell commands from some document, I have a function that combines a /read line from buffer/ and then send it to the currently running terminal: #+BEGIN_SRC emacs-lisp (defun ha-ssh-send-line () "Copy the contents of the current line in the current buffer, and call =ha-ssh-send= with it. After sending the contents, it returns to the current line." (interactive) ;; The function =save-excursion= doesn't seem to work... (let* ((buf (current-buffer)) (cmd-line (buffer-substring-no-properties (line-beginning-position) (line-end-position))) (trim-cmd (s-trim cmd-line))) (ha-ssh-send trim-cmd) ;; (sit-for 0.25) (pop-to-buffer buf))) #+END_SRC Let's have a quick way to bugger out of the terminal: #+BEGIN_SRC emacs-lisp (defun ha-ssh-exit (&optional window-name) "End the SSH session specified by WINDOW-NAME (or if not, the latest session)." (interactive) (unless (string-match-p "v?term" (buffer-name)) (unless window-name (setq window-name ha-latest-ssh-window-name)) (pop-to-buffer window-name)) (ignore-errors (term-send-eof)) (kill-buffer window-name) (delete-window)) #+END_SRC ** Editing Remote Files TRAMP, when it works, is amazing that we can give it a reference to a remote directory, and have =find-file= magically autocomplete. #+BEGIN_SRC emacs-lisp (defun ha-ssh-find-file (hostname) "Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=." (interactive (list (ha-ssh-choose-host))) (let ((tramp-ssh-ref (format "/ssh:%s:" hostname)) (other-window (when (equal current-prefix-arg '(4)) t))) (ha-ssh--find-file tramp-ssh-ref other-window))) (defun ha-ssh--find-file (tramp-ssh-ref &optional other-window) "Calls =find-file= after internally completing a file reference based on TRAMP-SSH-REF." (let ((tramp-file (read-file-name "Find file: " tramp-ssh-ref))) (if other-window (find-file-other-window tramp-file) (find-file tramp-file)))) #+END_SRC We can even edit it as root: #+BEGIN_SRC emacs-lisp (defun ha-ssh-find-root (hostname) "Constructs a ssh-based, tramp-focus, file reference, and then calls =find-file=." (interactive (list (ha-ssh-choose-host))) (let ((tramp-ssh-ref (format "/ssh:%s|sudo:%s:" hostname hostname)) (other-window (when (equal current-prefix-arg '(4)) t))) (ha-ssh--find-file tramp-ssh-ref other-window))) #+END_SRC ** OpenStack Interface Instead of making sure I have a list of remote systems already in the favorite hosts cache, I can pre-populate it with a call to OpenStack (my current VM system I'm using). These calls to the =openstack= CLI assume that the environment is already filled with the credentials. Hey, it is my local laptop ... We'll give =openstack= CLI a =--format json= option to make it easier for parsing: #+BEGIN_SRC emacs-lisp (use-package json) #+END_SRC Need a variable to hold all our interesting hosts. Notice I use the word /overcloud/, but this is a name I've used for years to refer to /my virtual machines/ that I can get a listing of, and not get other VMs that I don't own. #+BEGIN_SRC emacs-lisp (defvar ha-ssh-overcloud-cache-data nil "A vector of associated lists containing the servers in an Overcloud.") #+END_SRC If our cache data is empty, we could automatically retrieve this information, but only on the first time we attempt to connect. To do this, we'll =advice= the =ha-ssh-choose-host= function defined earlier: #+BEGIN_SRC emacs-lisp (defun ha-ssh-overcloud-query-for-hosts () "If the overcloud cache hasn't be populated, ask the user if we want to run the command." (when (not ha-ssh-overcloud-cache-data) (when (y-or-n-p "Cache of Overcloud hosts aren't populated. Retrieve hosts?") (call-interactively 'ha-ssh-overcloud-cache-populate)))) (advice-add 'ha-ssh-choose-host :before 'ha-ssh-overcloud-query-for-hosts) #+END_SRC We'll do the work of getting the /server list/ with this function: #+BEGIN_SRC emacs-lisp (defun ha-ssh-overcloud-cache-populate (cluster) (interactive (list (completing-read "Cluster: " '(devprod1 devprod501 devprod502)))) (message "Calling the `openstack' command...this will take a while. Grab a coffee, eh?") (let* ((command (format "openstack --os-cloud %s server list --no-name-lookup --insecure -f json" cluster)) (json-data (shell-command-to-string command))) (message "Call to openstack returned: %s" json-data) (setq ha-ssh-overcloud-cache-data (json-read-from-string json-data)) (cl-flet* ((ssh-tuple (entry) (cons (alist-get 'Name entry) (aref (thread-last entry (alist-get 'Networks) (alist-get 'cedev13)) 0)))) (setq ha-ssh-favorite-hostnames (mapcar #'ssh-tuple ha-ssh-overcloud-cache-data)))) (message "Call to =openstack= complete. Found %d hosts." (length ha-ssh-overcloud-cache-data))) #+END_SRC In case I change my virtual machines, I can repopulate that cache: #+BEGIN_SRC emacs-lisp (defun ha-ssh-overcloud-cache-repopulate () "Repopulate the cache based on redeployment of my overcloud." (interactive) (setq ha-ssh-overcloud-cache-data nil) (call-interactively 'ha-ssh-overcloud-cache-populate)) #+END_SRC The primary interface: #+BEGIN_SRC emacs-lisp (defun ha-ssh-overcloud (hostname) "Log into an overcloud host given by HOSTNAME. Works better if you have previously run =ssh-copy-id= on the host. Remember, to make it behave like a real terminal (instead of a window in Emacs), hit =C-c C-k=." (interactive (list (ha-ssh-choose-host))) (when (not (string-match-p "\." hostname)) (setq hostname (format "%s.%s" hostname (getenv "OS_PROJECT_NAME")))) (let ((window-label (or (-some->> ha-ssh-favorite-hostnames (rassoc hostname) car) hostname))) (ha-ssh hostname window-label) (sit-for 1) (ha-ssh-send "sudo -i") (ha-ssh-send (format "export PS1='\\[\\e[34m\\]%s\\[\e[m\\] \\[\\e[33m\\]\\$\\[\\e[m\\] '" window-label)) (ha-ssh-send "clear"))) #+END_SRC * Keybindings This file, so far, as been good-enough for a Vanilla Emacs installation, but to hook into Doom's leader for some sequence binding, this code isn't: #+BEGIN_SRC emacs-lisp (ha-leader "a e" '("eshell" . eshell-here) "a E" '("top eshell" . eshell-project) "a s" '(:ignore t :which-key "ssh") "a s v" '("vterm" . vterm) "a s o" '("overcloud" . ha-ssh-overcloud) "a s l" '("local shell" . ha-shell) "a s s" '("remote shell" . ha-ssh) "a s q" '("quit shell" . ha-ssh-exit) "a s f" '("find-file" . ha-ssh-find-file) "a s r" '("find-root" . ha-ssh-find-root)) #+END_SRC * Technical Artifacts :noexport: Provide a name so we can =require= the file: #+BEGIN_SRC emacs-lisp :exports none (provide 'ha-remoting) ;;; ha-remoting.el ends here #+END_SRC Before you can build this on a new system, make sure that you put the cursor over any of these properties, and hit: ~C-c C-c~ #+DESCRIPTION: A literate configuration for accessing remote systems. #+PROPERTY: header-args:sh :tangle no #+PROPERTY: header-args:emacs-lisp :tangle yes #+PROPERTY: header-args :results none :eval no-export :comments no mkdirp yes #+OPTIONS: num:nil toc:nil todo:nil tasks:nil tags:nil date:nil #+OPTIONS: skip:nil author:nil email:nil creator:nil timestamp:nil #+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js