Editing Data Files

Table of Contents

A literate programming file for configuring Emacs to edit files of data.

Introduction

Once upon a time, I gave a talk to EmacsConf 2019, about an interesting idea I called emacs-piper. I still like the idea of sometimes editing an Emacs buffer on the entire contents, as if it were a data file. This file contains what I feel are the best functions for that… oh, and a leader to call it (instead of the original Hydra). 

(ha-leader
  "d"   '(:ignore t :which-key "data")
  "d |" '("pipe to shell"  . replace-buffer-with-shell-command)
  "d r" '("replace buffer"  . ha-vr-replace-all)
  "d y" '("copy to clipboard" . ha-yank-buffer-contents))

Global Replacements

The string replacement functions operate at the current point, which means I need to jump to the beginning before calling something like vr/replace. 

(defun ha-vr-replace-all (regexp replace start end)
  "Regexp-replace entire buffer with live visual feedback."
  (interactive
   (vr--interactive-get-args 'vr--mode-regexp-replace 'vr--calling-func-replace))
  (vr/replace regexp replace (point-min) (point-max)))

Line-Oriented Functions

These functions focus on the data in the buffer as a series of lines: 

(ha-leader
  "d l"   '(:ignore t :which-key "on lines")
  "d l d" '("flush lines"    . flush-lines)
  "d l k" '("keep lines"     . keep-lines)
  "d l s" '("sort lines"     . ha-sort-lines)
  "d l f" '("sort fields"    . ha-sort-fields)
  "d l n" '("sort field num" . ha-sort-fields-numerically)
  "d l r" '("reverse lines"  . ha-reverse-lines)
  "d l u" '("unique lines"   . delete-duplicate-lines)
  "d l b" '("flush blanks"   . flush-blank-lines))

One issue I have is keep-lines operate on the lines starting with the point, not on the entire buffer. Let’s fix that: 

(defun call-function-at-buffer-beginning (orig-fun &rest args)
  "Call ORIG-FUN after moving point to beginning of buffer.
Point restored after completion. Good for advice."
  (save-excursion
    (goto-char (point-min))
    (apply orig-fun args)))

(advice-add 'keep-lines :around #'call-function-at-buffer-beginning)
(advice-add 'flush-lines :around #'call-function-at-buffer-beginning)

The sort-lines is useful, but insists on an active region. Let’s make a collection of data-focused versions that work on both a region (if it is active) or the entire buffer, regardless of the position of the cursor. 

(dolist (tuple '((ha-sort-lines sort-lines)
                 (ha-sort-fields sort-fields)
                 (ha-sort-fields-numerically sort-numeric-fields)))
  (cl-destructuring-bind (func orig-func) tuple
    (eval `(defun ,func (prefix)
             ,(format "Call `%s' with all lines in a buffer or region (if active).
Passes PREFIX to the function." orig-func)
             (interactive "P")
             (save-excursion
               (if (region-active-p)
                   (,orig-func prefix (region-beginning) (region-end))
                 (,orig-func prefix (point-min) (point-max))))))))

Getting rid of blank lines seems somewhat useful: 

(defun flush-blank-lines ()
  "Delete all empty lines in the buffer. See `flush-lines'."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (flush-lines (rx line-start (zero-or-more space) line-end))))

Table-Oriented Functions

These functions focus on the data in the buffer as a table consisting of columns and rows of some sort. 

(ha-leader
  "d t"   '(:ignore t :which-key "on tables")
  "d t f" '("sort by columns"    . ha-sort-fields)
  "d t n" '("sort by columns numerically" . ha-sort-fields-numerically)
  "d t k" '("keep columns" . keep-columns)
  "d t f" '("flush columns" . flush-columns))

Each of the table functions require a table separator (for instance the | character) and often the columns to operate on. The keep-columns removes all text, except for indexed text between the separator: 

(defun keep-columns (separator columns)
  "Keep tabular text columns, deleting the rest in buffer or region.
Defined columns as text between SEPARATOR, not numerical
position. Note that text _before_ the separator is column 0.

For instance, given the following table:

   apple : avocado : apricot
   banana : blueberry : bramble
   cantaloupe : cherry : courgette : cucumber
   data : durian

Calling this function with a `:' character, as columns: `0, 2'
results in the buffer text:

   apple : apricot
   banana : bramble
   cantaloupe : courgette
   data : durian"
  (interactive "sSeparator: \nsColumns to Keep: ")
  (operate-columns separator columns t))

The flush-columns is similar, except that is deletes the given columns. 

(defun flush-columns (separator columns)
  "Delete tabular text columns in buffer or region.
Defined columns as text between SEPARATOR, not numerical
position. Note that text _before_ the separator is column 0.

For instance, given the following table:

   apple : avocado : apricot
   banana : blueberry : bramble
   cantaloupe : cherry : courgette : cucumber
   data : durian

Calling this function with a `:' character, as columns: `1'
(remember the colums are 0-indexed),
results in the buffer text:

   apple : avocado : apricot
   banana : blueberry : bramble
   cantaloupe : cherry : courgette : cucumber
   data : durian

   apple : apricot
   banana : bramble
   cantaloupe : courgette
   data : durian"
  (interactive "sSeparator: \nsColumns to Delete: ")
  (operate-columns separator columns nil))

Both functions are similar, and their behavior comes from operate-columns, which walks through the buffer, line-by-line: 

(defun operate-columns (separator columns-str keep?)
  "Call `operate-columns-on-line' for each line in buffer.
First, convert string COLUMNS-STR to a list of number, then
search for SEPARATOR."
  (let ((columns (numbers-to-number-list columns-str)))
    (save-excursion
      (when (region-active-p)
        (narrow-to-region (region-beginning) (region-end)))
      (goto-char (point-min))
      (while (re-search-forward (rx (literal separator)) nil t)
        (operate-columns-on-line separator columns t)
        (next-line)))))

For each line, the operate-columns calls this function: 

(defun operate-columns-on-line (separator columns keep?)
  "Replace current line after keeping or deleting COLUMNS.
Keep the COLUMNS if KEEP? is non-nil, delete otherwise.
Defined columns as the text between SEPARATOR."
  (cl-labels ((keep-oper (idx it) (if keep?
                                      (when (member idx columns) it)
                                    (unless (member idx columns) it))))
    (let* ((start (line-beginning-position))
           (end (line-end-position))
           (line (buffer-substring start end))
           (parts (thread-last (split-string line separator)
                               (--map-indexed (keep-oper it-index it))
                               (-remove 'null)))
           (nline (string-join parts separator)))
      (delete-region start end)
      (insert nline))))

I like the idea of the shell command, cut, where you can have an arbitrary character as a separator, and then either delete or keep the data between them, as columns. But I need a function that can convert a string of “columns”, for instance "1, 4-7 9" to an list of numbers, like '(1 4 5 6 7 9): 

(defun numbers-to-number-list (input)
  "Convert the string, INPUT, to a list of numbers.
For instance: `1, 4-7 9' returns `(1 4 5 6 7 9)'"
  (let* ((separator (rx (* space) (or "," space) (* space)))
         (dashed    (rx (* space) "-" (* space)))
         (ranged    (rx (group (+ digit)) (regexp dashed) (group (+ digit))))
         (str-list  (split-string input separator t)))
    (--reduce-from (append acc (if (string-match ranged it)
                               (number-sequence
                                (string-to-number (match-string 1 it))
                                (string-to-number (match-string 2 it)))
                             (list (string-to-number it))))
              () str-list)))

Does this work? 

(ert-deftest numbers-to-number-list-test ()
  (should (equal (numbers-to-number-list "2") '(2)))
  (should (equal (numbers-to-number-list "1, 2 3") '(1 2 3)))
  (should (equal (numbers-to-number-list "1, 4-7 9") '(1 4 5 6 7 9))))

The sort-fields function does a good job if the table is space separated, but if we separate by some other character(s), it doesn’t work. Can we write a function that does this? Here we make a helper to some regular expression for the sort-regexp-fields function. 

(defun ha-sort-table-by-column (separator field)
  "Sort the active region or entire buffer by column, FIELD.
Columns are denoted by a regular expression, SEPARATOR, which
could be a single character. For instance, given a buffer:

 d, a, c, b
 x, y
 e, f, g
 i, m, a, o

Calling this function with a `,' separator, and `2' for the
column, would result in:

 d, a, c, b
 e, f, g
 i, m, a, o
 x, y"
  (interactive "sSeparator: \nnSorting field: ")
  ;; Create a regular expression of grouped fields, separated
  ;; by the separator sequence, for commas, this would be e.g.
  ;;    \\(.*\\),\\(.*\\),\\(.*\\),\\(.*\\)
  (let* ((rx-list (mapconcat (lambda (x) (rx (group (zero-or-more any))))
                             (number-sequence 1 field)
                             separator))
         ;; Prepend the beginning of line to the regular expression:
         (regexp (concat (rx bol) rx-list))
         (start  (if (region-active-p) (region-beginning) (point-min)))
         (end    (if (region-active-p) (region-end)       (point-max))))
    (save-excursion
      (sort-regexp-fields nil regexp (format "\\%d" field) start end))))

Buffer-Oriented Functions

If there is no specific function, but you can think of a shell command that will work, then 

(defun replace-buffer-with-shell-command (command)
  "Replaces the contents of the buffer, or the contents of the
selected region, with the output from running an external
executable, COMMAND.

This is a wrapper around `shell-command-on-region'."
  (interactive "sCommand: ")
  (save-excursion
    (save-restriction
      (when (region-active-p)
        (narrow-to-region (region-beginning) (region-end)))
      (shell-command-on-region (point-min) (point-max) command nil t))))