Literate Programming with Org

Introduction
Snippets
- Surround with Org Block
- Split an Org Block
Visit Tangled File
Navigating Code Blocks
Evaluating Code
Finding Code
Keybindings

A literate programming file for literate programming in Emacs Org Files.

Introduction

I do a lot of literate programming using capabilities found in the Org project. Over the years, I’ve smoothed some of the rough edges by writing supporting functions, collected below.

What are the advantage of literate programming? I listed some in my essay and video about my literate devops ideas, but a brief recap:

ambiguous and/or complicated ideas can be reasoned about in prose before code
links to ideas, code snippets and supporting projects are more accessible than in comments
diagrams written in PlantUML, Graphviz, or Pikchr maintained and displayed with code
full access to org features, like clocks, task lists and agendas
mixing languages in the same file, e.g. shell instructions for installing supporting modules before the code that uses that module

If literate programming is so great, why doesn’t everyone do it? Most of the advantages listed above are advantages only because we are using Emacs and Org Mode, and getting teams to adopt this requires changing editors and workflows. Therefore, literate programming is a solo endeavor.

Since we are using Emacs, the downsides of using a literate approach (even for personal projects) can be minimized.

Note: What follows is advanced LP usage. I would recommend checking out my Introduction to LP in Org before venturing down this essay.

Snippets

Surround with Org Block

This simple change allows me to highlight any text, and surround it in an org block. We

# name: surround-org-block
# --
#+BEGIN_${1:SRC} ${2:emacs-lisp} $0
`yas-selected-text`
#+END_$1

Without a key (that is intentional), I can call the snippet with C-c & C-s (ugh).

Split an Org Block

What is the language or major-mode associated with the “current” block?

(defun ha-org-block-language ()
  "Return the language associated with the current block."
  (save-excursion
    (re-search-backward (rx bol (zero-or-more space)
                            "#+begin_src"
                            (one-or-more space)
                            (group (one-or-more (not space)))))
    (match-string 1)))

Actually, I probably want all the parameters (if any) to come along for the ride:

(defun ha-org-block-language-and ()
  "Return language and parameters of the current block."
  (save-excursion
    (re-search-backward (rx bol (zero-or-more space)
                            "#+begin_src"
                            (one-or-more space)
                            (group (one-or-more any))))
    (match-string 1)))

This allows us to now split a block into two parts:

,# -*- mode: snippet -*-
,# name: split-org-block
,# key: #split
,# --
#+END_SRC
`yas-selected-text`
#+BEGIN_SRC `(ha-org-block-language-and)`

Visit Tangled File

After tangling a file, you might want to take a look at it. But a single org file can tangle out to more than one file, so how to choose? Each block could have its one file, as would the particular language-specific blocks in a section or the entire file. I don’t care to be too pendantic here, because I often know the name of the file.

Perhaps a little function to map a language to a file name:

(defun ha-org-babel-tangle-default (language)
  "Return the default tangled file based on LANGUAGE."
  (format "%s.%s"
          (file-name-sans-extension (buffer-file-name))
          (pcase language
            ("emacs-lisp" "el")
            ("elisp" "el")
            ("python" "py")
            ("ruby" "rb")
            ;; ...
            ("bash" "sh")
            ("sh" "sh")
            ("shell" "sh"))))

This function returns the name of the tangled file based on either the block or a default name based on the file name:

(defun ha-org-babel-tangle-file-name ()
  "Return _potential_ file name or Org source.
If nearest block has a `:tangle' entry, return that.
If block doesn't have `:tangle' (or set to `yes'),
return the filename based on the block's language."
  (save-excursion
    (org-previous-block 1)
    (when (looking-at
           (rx "#+begin_src"
               (one-or-more space)
               (group (one-or-more (not space)))
               (one-or-more space) (zero-or-more any)
               (optional
                (group ":tangle" (one-or-more space)
                       (optional "\"")
                       (group (one-or-more any))
                       (optional "\"")))))
      (let ((language (match-string 1))
            (filename (match-string 3)))
        (if (or (null filename) (equal filename "yes") (equal filename "true"))
            (ha-org-babel-tangle-default language)
          filename)))))

And this function calls find-file on the tangled file (if found):

(defun ha-org-babel-tangle-visit-file ()
  "Attempt to call `find-file' on Org's tangled file."
  (interactive)
  (let ((tangled-file (ha-org-babel-tangle-file-name)))
    (if (file-exists-p tangled-file)
        (find-file tangled-file)
      (message "Looking for %s, which doesn't exist." tangled-file))))

Note: You can return to the original Org file with org-babel-tangle-jump-to-org, if you set the tangle comments to link, as in this global property:

#+PROPERTY: header-args:emacs-lisp :tangle yes :comments link

Navigating Code Blocks

I’ve been using Oleh Krehel’s (abo-abo) Avy project to jump around the screen for years, and I just learned that I can wrap the avy-jump function to provide either/or regular expression and action to perform.

For instance, the following function can be used to quickly select a source code block, and jump to it:

(defun avy-jump-org-block ()
  "Jump to org block using Avy subsystem."
  (interactive)
  (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
            :action 'goto-char)
  ;; Jump _into_ the block:
  (forward-line))

I need to take advantage of this feature more.

Evaluating Code

Hitting C-c C-c in a source code block evaluates the code. Simple, sure, but the following enhancements make this more accessible.

Evaluating a Block

At times I would like to jump to a particular block, evaluate the code, and jump back. This seems like a great job for the avy project. The avy-jump function takes a regular expression of text in the frame (which means you can specify text in other windows), and highlights each match. Normally, selecting a match moves the cursor to that match, the avy-jump accepts a function to execute instead:

(defun org-babel-execute-src-block-at-point (&optional point)
  "Call `org-babel-execute-src-block' at POINT."
  (save-excursion
    (goto-char point)
    (org-babel-execute-src-block)))

(defun avy-org-babel-execute-src-block ()
  "Call `org-babel-execute-src-block' on block given by Avy.
Use Avy subsystem to select a visible Org source code block,
e.g. `#+begin_src', and then executes the code without moving
the point."
  (interactive)
  (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
            :action 'org-babel-execute-src-block-at-point))

In this case, avy-org-babel-execute-src-block highlights all visible blocks on the frame, with a letter on each. Selecting the letter, evaluates that block without moving the cursor.

TODO Screenshot of multiple highlighted blocks.

Evaluating a Section

A trick to org-babel-tangle, is that it tangles what Emacs shows, that is, it tangles visible code blocks after narrowing to the current org section. This means, we can call org-narrow-to-subtree to temporary hide everything in the org file except the current heading, evaluate all blocks in the “now visible” buffer, and then widen:

(defun org-babel-execute-subtree (prefix)
  "Execute all Org source blocks in current subtree.
The PREFIX is passed to `org-babel-execute-buffer'."
  (interactive "P")
  (save-excursion
    (org-narrow-to-subtree)
    (org-babel-execute-buffer prefix)
    (widen)))

Editing a Block

Why navigate to a block, solely to focus on that block in a dedicated buffer, when we can take advantage of the avy-jump and edit any visible block?

(defun org-babel-edit-src-block-at-point (&optional point)
  "Call `org-babel-execute-src-block' at POINT."
  (save-excursion
    (goto-char point)
    (org-edit-src-code)))

(defun avy-org-babel-edit-src-block ()
  "Call `org-edit-src-code' on block given by Avy.
Use Avy subsystem to select a visible Org source code block,
e.g. `#+begin_src', and then executes the code without moving
the point."
  (interactive)
  (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
            :action
            'org-babel-edit-src-block-at-point))

Finding Code

One of the issues with literate programming is not being able to use the same interface for moving around code when the source code is in org files.

XRef Interface

The Emacs interface for jumping to function definitions and variable declarations is called xref (see this great article for an overview of the interface). I think it would be great to be able, even within the prose of an org file, to jump to the definition of a function that is defined in an org file.

Definitions: To jump to the line where a macro, function or variable is defined.
References: To get a list of all calls or usage of a symbol, but only within code blocks.
Apropos: To get a list of all references, even within org-mode prose.

In a normal source code file, you know the language, so you have way of figuring out what a symbol is and how it could be defined in that language. In org files, however, one can use multiple languages, even in the same file.

In the code that follows, I’ve made an assumption that I will primarily use this xref interface for Emacs Lisp code, however, it wouldn’t take much (a single regular expression) to convert to another language.

Taking a cue from dumb-jump, I’ve decided to not attempt to build any sort of tag interaction, but instead, call ripgrep. I love that its –-json option outputs much more parseable text.

Symbols

I wrote the ha-literate-symbol-at-point function as an attempt at being clever with figuring out what sort of symbol references we would want from an org file. I assume that a symbol may be written surrounded by ~ or = characters (for code and verbatim text), as well as in quotes or braces, etc.

While the goal is Emacs Lisp (and it mostly works for that), it will probably work for other languages as well.

(defun ha-literate-symbol-at-point ()
  "Return an alphanumeric sequence at point.
Assuming the sequence can be surrounded by typical
punctuation found in `org-mode' and markdown files."
  (save-excursion
    ;; Position point at the first alnum character of the symbol:
    (cond ((looking-at (rx (any "=~({<\"'“`") alnum))
           (forward-char))
          ;; Otherwise go back to get "inside" a symbol:
          ((not (looking-at (rx alnum)))
           (re-search-backward (rx alnum))))

    ;; Move point to start and end of the symbol:
    (let ((start (progn (skip-chars-backward "a-zA-Z0-9_-") (point)))
          (end   (progn (skip-chars-forward "?a-zA-Z0-9_-") (point))))
      (buffer-substring-no-properties start end))))

Examples of references in an Org file that should work:

ha-literate-symbol-at-point
“ha-literate-symbol-at-point”
`ha-literate-symbol-at-point`

This magical incantation connects our function to Xref with an org backend:

(cl-defmethod xref-backend-identifier-at-point ((_backend (eql org)))
  (ha-literate-symbol-at-point))

Calling ripgrep

This helper function does the work of calling ripgrep, parsing its output, and filtering only the matches line. Yes, an interesting feature of rg is that it spits out a sequence of JSON-formatted text, so we can use seq-filter to grab lines that represent a match, and seq-map to “do the work”. Since we have a couple of ways of doing the work, we pass in a function, processor, which, along with transforming the results, could spit out nulls, so the seq-filter with the identity function eliminates that.

(defun ha-literate--ripgrep-matches (processor regex &optional symb)
  "Return list of running PROCESSOR of `rg' matches from REGEX.
PROCESSOR is called with an assoc-list of the JSON output from
the call to ripgrep."
  (let* ((default-directory (if (project-current)
                                (project-root (project-current))
                              default-directory))
         (search-str (rxt-elisp-to-pcre regex))
         (command (format "rg --json --type org '%s'" search-str)))

    ;; (message "Literate xref Calling: %s" command)
    (message "Searching `%s' %s..." default-directory
             (if symb (format "for `%s' " symb) ""))
    (thread-last command
                 (shell-command-to-list)
                 (seq-map 'ha-literate--parse-rg-line)
                 (seq-filter 'ha-literate--only-matches)
                 (seq-map processor)
                 ;; Remove any nulls from the list:
                 (seq-filter 'identity))))

Note: the processor function creates an xref object, described below. See ha-literate--process-rg-line.

The output from ripgrep goes through a couple of transformation functions listed here:

(defun ha-literate--parse-rg-line (line)
  "Process LINE as a JSON object with `json-parse-string'."
  (json-parse-string line :object-type 'alist
                          :array-type 'list))

(defun ha-literate--only-matches (json-data)
  "Return non-nil if JSON-DATA is an alist with key `type' and value `match'."
  (string-equal "match" (alist-get 'type json-data)))

Since our ripgrep searches from the project root, but xref wants to make file references relative to the buffer that is calling it, we need to make some changes:

(defun ha-literate-make-xref-file (filepath)
  "Return FILEPATH relative to current buffer's file."
  (let ((abspath (expand-file-name filepath
                                   (if (project-current)
                                       (project-root (project-current))
                                     default-directory)))
        (relative-to (file-name-parent-directory (buffer-file-name))))
    (file-relative-name abspath relative-to)))

Let’s test this function:

(ert-deftest ha-literate-make-xref-file-test ()
  ;; Current directory
  (should (equal (ha-literate-make-xref-file "ha-display.org")
                 "ha-display.org"))
  ;; Subdirectory
  (should (equal (ha-literate-make-xref-file "elisp/beep.el")
                 "elisp/beep.el"))
  (should (equal (ha-literate-make-xref-file "~/foo/bar.org")
                 "../../foo/bar.org")))

Definitions

As mentioned above, let’s assume we can use ripgrep to search for definitions in Lisp. I choose that because most of my literate programming is in Emacs Lisp. This regular expression should work with things like defun and defvar, etc. as well as use-package, allowing me to search for the definition of an Emacs package:

(defun ha-literate-definition-rx (symb)
  "Return a regular expression to search for definition of SYMB."
  (rx "("
      (or "use-package"
          (seq ; Match both defun and cl-defun:
           (optional "cl-")
           "def" (1+ (not space))))
      (one-or-more space)
      (literal symb)
      word-boundary))

(defun ha-literate-definition (symb)
  "Return list of `xref' objects of SYMB location in org files.
The location is based on a regular expression starting with
`(defxyz SYMB' where this can be `defun' or `defvar', etc."
  (ha-literate--ripgrep-matches 'ha-literate--process-rg-line
                                (ha-literate-definition-rx symb)
                                symb))

The work of processing a match for the ha-literate-definition function. It calls xref-make to create an object for the Xref system. This takes two parameters, the text and the location. We create a location with xref-make-file-location.

(defun ha-literate--process-rg-line (rg-data-line)
  "Return an `xref' structure based on the contents of RG-DATA-LINE.
The RG-DATA-LINE is a convert JSON data object from ripgrep.
The return data comes from `xref-make' and `xref-make-file-location'."
  (when rg-data-line
    (let-alist rg-data-line
      ;; (message "xref-make %s" .data.path.text)
      (xref-make .data.lines.text
                 (xref-make-file-location
                  ;; Relative filename:
                  (ha-literate-make-xref-file .data.path.text)
                  ;; Line number:
                  .data.line_number
                  ;; Column: Icky to parse:
                  (thread-last
                    (first .data.submatches)
                    (alist-get 'start)))))))

I like the use of let-alist where I can access the parsed output from JSON via variables, like .data.path.text.

We connect this function to the xref-backend-definitions list, so that it can be called when we type something like M-.:

(cl-defmethod xref-backend-definitions ((_backend (eql org)) symbol)
  (ha-literate-definition symbol))

Apropos

The apropos approach is anything, so the regular expression here is just the symbol, and we can re-use our processor:

(defun ha-literate-apropos (symb)
  "Return an `xref' object for SYMB location in org files.
The location is based on a regular expression starting with
`(defxyz SYMB' where this can be `defun' or `defvar', etc."
  (ha-literate--ripgrep-matches 'ha-literate--process-rg-line
                                (rx word-boundary
                                    (literal symb)
                                    word-boundary)))

And this to hook it up:

(cl-defmethod xref-backend-apropos ((_backend (eql org)) symbol)
  (ha-literate-apropos symbol))

References

While traditionally, -apropos can reference symbols in comments and documentation, searching for references tend to be calls and whatnot. What does that mean in the context of an org file? I’ve decided that references should only show symbols within org blocks.

How do we know we are inside an org block?

I call ripgrep twice, once to get all the begin_ and end_src lines and their line numbers. The second ripgrep call gets the references.

(defun ha-literate-references (symb)
  "Return list of `xref' objects for SYMB location in org files.
The location is limited only references in org blocks."
  ;; First, get and store the block line numbers:
  (ha-literate--block-line-numbers)
  ;; Second, call `rg' again to get all matches of SYMB:
  (ha-literate--ripgrep-matches 'ha-literate--process-rg-block
                                (rx word-boundary
                                    (literal symb)
                                    word-boundary)))

Notice for this function, we need a new processor that limits the results to only matches between the beginning and ending of a block, which I’ll describe later.

The ha-literate--block-line-numbers returns a hash where the keys are files, and the value is a series of begin/end line numbers. It calls ripgrep, but has a new processor.

(defun ha-literate--block-line-numbers ()
  "Call `ripgrep' for org blocks and store results in a hash table.
See `ha-literate--process-src-refs'."
  (clrhash ha-literate--process-src-refs)
  (ha-literate--ripgrep-matches 'ha-literate--process-src-blocks
      (rx line-start (zero-or-more blank)
                          "#+" (or "begin" "end") "_src")))

And the function to process the output simply attempts to connect the begin_src with the end_src lines. In true Emacs Lisp fashion (where we can’t easily, lexically nest functions), we use a global variable:

(defvar ha-literate--process-src-refs
  (make-hash-table :test 'equal)
  "Globabl variable storing org-mode's block line numbers.
The key in this table is a file name, and the value is a list of
line numbers marking #+begin_src and #+end_src.")

(defvar ha-literate--process-begin-src nil
  "Global variable last `#+begin_src' line number.")

(defun ha-literate--process-src-blocks (rg-data-line)
  "Return nil if RG-DATA-LINE contains a begin_src entry.
Otherwise return a list of previous begin_src, and the
current end_src line numbers."
  (let-alist rg-data-line
    (puthash .data.path.text ; filename is the key
             (append
              (gethash .data.path.text ha-literate--process-src-refs)
              (list .data.line_number))
             ha-literate--process-src-refs)))

With a collection of line numbers for all org-blocks in all org files in our project, we can process a particular match from ripgrep to see if the match is within a block. Since the key is a file, and .data.path.text is the filename, that part is done, but we need a helper to walk down the list.

(defun ha-literate--process-rg-block (rg-data-line)
  "Return an `xref' structure from the contents of RG-DATA-LINE.
Return nil if the match is _not_ with org source blocks.
Note that the line numbers of source blocks should be filled
in the hashmap, `ha-literate--process-src-refs'."
  (let-alist rg-data-line
    (let ((line-nums (thread-first .data.path.text
                                   (gethash ha-literate--process-src-refs)
                                   ;; Turn list into series of tuples
                                   (seq-partition 2))))
      (when (ha-literate--process-in-block .data.line_number line-nums)
        (ha-literate--process-rg-line rg-data-line)))))

(defun ha-literate--process-in-block (line-number line-numbers)
  "Return non-nil if LINE-NUMBER is inclusive in LINE-NUMBERS.
The LINE-NUMBERS is a list of two element lists where the first
element is the starting line number of a block, and the second
is the ending line number."
  (when line-numbers
    (let ((block-lines (car line-numbers)))
      (if (and (> line-number (car block-lines))
               (< line-number (cadr block-lines)))
          (car block-lines)
        (ha-literate--process-in-block line-number (cdr line-numbers))))))

The helper function, ha-literate--process-in-block is a recursive function that takes each tuple and sees if line-number is between them. If it isn’t between any tuple, and the list is empty, then we return nil to filter that out later.

Let’s connect the plumbing:

(cl-defmethod xref-backend-references ((_backend (eql org)) symbol)
  (ha-literate-references symbol))

Whew! This is pretty cool to jump out my literate code base as if it were actual .el files.

Identifier Completion Table

Need the completion table before we can find the references. It actually doesn’t even need to return anything purposeful:

(defun ha-literate-completion-table ())

Now we hook this up to the rest of the system, and the xref is now complete:

(cl-defmethod xref-backend-identifier-completion-table ((_backend (eql org)))
  (ha-literate-completion-table))

Activation of my Literate Searching

To finish the connections, we need to create a hook that I only allow to turn on with org files:

(defun ha-literate-xref-activate ()
  "Function to activate org-based literate backend.
Add this function to `xref-backend-functions' hook."
  (when (eq major-mode 'org-mode)
    'org))

;; Add this hook to the beginning, as we want to call our
;; backend reference before dumb-jump:
(add-hook 'xref-backend-functions #'ha-literate-xref-activate -100)

At this point, we can jump to functions and variables that I define in my org file, or even references to standard symbols like xref-make or xref-backend-functions.

I can jump around my literate code as if they were .el files. I may want to think about expanding the definitions to figure out the language of the destination.

Noweb References

A noweb definition, e.g. <<something-something>> should jump to the #name definition.

Since ripgrep is pretty fast, I’ll call it instead of attempting to build a CTAGS table. Oooh, the rg takes a —json option, which makes it easier to parse.

(defun ha-org-noweb-block-jump (str pos)
  "Go to a literate org file containing a symbol, STR.
The POS is ignored."
  ;; Sometimes I wrap a function name in `=' characters, and these should be removed:
  (when (string-match (rx "<<" (group (one-or-more any)) ">>") str)
    (setq str (match-string 1 str)))
  (ignore-errors
    (let* ((default-directory (project-root (project-current)))
           (command (format "rg --ignore-case --json '#\\+name: +%s' *.org" str))

           (results (thread-last command
                                 shell-command-to-list
                                 second
                                 json-parse-string))
           (file    (thread-last results
                                 (gethash "data")
                                 (gethash "path")
                                 (gethash "text")))
           (line    (thread-last results
                                 (gethash "data")
                                 (gethash "line_number"))))
      (find-file file)
      (goto-line line))))

(when (fboundp 'evil-goto-definition-functions)
  (add-to-list 'evil-goto-definition-functions 'ha-org-noweb-block-jump))

Searching by Header

As large literate programming projects grow, I refine, re-organize and refactor content. I don’t always remember where I put particular code. For instance, in my Emacs configuration, did I configure eww, in my default config file, or did I move it somewhere? Originally, after loading the file, I could issue a call to consult-imenu or org-consult-heading to get to the right location, but that assumes I have the correct file loaded.

The following section shows some code to use the fuzzy matching features of Orderless, to choose a headline in any of my Org files in a project, and then load that file and jump to that headline. The interface is ha-hamacs-edit-file-heading, and the supporting functions begin with ha-hamacs-edit-:

(defun ha-hamacs-edit-file-heading (&optional project-root)
  "Edit a file based on a particular heading.
After presenting list of headings from all Org files,
it loads the file, and jumps to the line number where
the heading is located."
  (interactive)
  (let* ((default-directory (or project-root (project-root (project-current))))
         (file-headings (ha-hamacs-edit--file-heading-list))
         (file-choice   (completing-read "Edit Heading: " file-headings))
         (file-tuple    (alist-get file-choice file-headings
                                   nil nil 'string-equal)))
    (find-file (first file-tuple))
    (goto-line (second file-tuple))))

This function collects all possible headers by issuing a call to ripgrep, which returns something like:

ha-applications.org:29:* Git and Magit
ha-applications.org:85:** Git Gutter
ha-applications.org:110:** Git Delta
ha-applications.org:136:** Git with Difftastic
...
"ha-applications.org:385:* Web Browsing
ha-applications.org:386:** EWW
...

We then filter out non-useful headers (with ha-hamcs-edit—filter-heading), and convert the headlines with ha-hamcs-edit—process-entry to be more presentable:

(defun ha-hamacs-edit--file-heading-list ()
  "Return list of lists of headlines and file locations.
This is found by calling `ripgrep' in the `default-directory'.
Using the output from the shell command, `ha-hamacs-edit-ripgrep-headers',
it parses and returns something like:

'((\"Applications∷ Git and Magit\" \"ha-applications.org\" 29)
  (\"Applications∷ Git and Magit ﹥ Git Gutter\" \"ha-applications.org\" 85)
  (\"Applications∷ Git and Magit ﹥ Git Delta\" \"ha-applications.org\" 110)
  (\"Applications∷ Git and Magit ﹥ Time Machine\" \"ha-applications.org\" 265)
  ...)"
  (thread-last ha-hamacs-edit-ripgrep-headers
               (shell-command-to-list)
               ;; Let's remove non-helpful, duplicate headings,
               ;; like Introduction:
               (seq-remove 'ha-hamacs-edit--filter-heading)
               ;; Convert the results into both a displayable
               ;; string as well as the file and line structure:
               (seq-map 'ha-hamacs-edit--process-entry)))

As the above function’s documentation string claims, I create a list that contains the data structure necessary for completing-read as well as the information I need to load/jump to a position in the file. This is a three-element list of the headline, filename and line number for each entry:

'(("Applications∷ Git and Magit" "ha-applications.org" 29)
  ("Applications∷ Git and Magit ﹥ Git Gutter" "ha-applications.org" 85)
  ("Applications∷ Git and Magit ﹥ Git Delta" "ha-applications.org" 110)
  ("Applications∷ Git and Magit ﹥ Time Machine" "ha-applications.org" 265)
  ("Applications∷ Git and Magit ﹥ Gist" "ha-applications.org" 272)
  ("Applications∷ Git and Magit ﹥ Forge" "ha-applications.org" 296)
  ("Applications∷ Git and Magit ﹥ Pushing is Bad" "ha-applications.org" 334)
  ("Applications∷ Git and Magit ﹥ Github Search?" "ha-applications.org" 347)
  ("Applications∷ ediff" "ha-applications.org" 360)
  ("Applications∷ Web Browsing" "ha-applications.org" 385)
  ("Applications∷ Web Browsing ﹥ EWW" "ha-applications.org" 386)
  ;; ...
  )

We’ll use this shell command to call ripgrep to search my collection of org files:

(defvar ha-hamacs-edit-ripgrep-headers
  (concat "rg"
          " --no-heading"
          " --line-number"
          ;; " --max-depth 1"
          " -e '^\\*+ '"
          " *.org")
  "A ripgrep shell call to search my headers.")

Not every header should be a destination, as many of my org files have duplicate headlines, like Introduction and Technical Artifacts, so I can create a regular expression to remove or flush entries:

(defvar ha-hamacs-edit-flush-headers
  (rx "*" (one-or-more space)
      (or "Introduction"
          "Install"
          "Overview"
          "Summary"
          "Technical Artifacts"))
  "Regular expression matching headers to purge.")

Note: This variable should be set in the .dir-locals.el for a particular project, as in:

((org-mode . ((ha-hamacs-edit-flush-headers .
               "\\*[[:space:]]+\\(?:Background\\|Summary\\)"))))

And this next function is callable by the filter function, it uses the regular expression and returns true (well, non-nil) if the line entry given, rg-input, should be removed:

(defun ha-hamacs-edit--filter-heading (rg-input)
  "Return non-nil if we should remove RG-INPUT.
These are headings with typical, non-unique entries,
like Introduction and Summary."
  (string-match ha-hamacs-edit-flush-headers rg-input))

The seq-map needs to take each line from the ripgrep call and convert it to a list that I can use for the completing-read prompt. I love the combination of seq-let and s-match from Magnar’s String library. The built-in function, string-match returns the index in the string where the match occurs, and this is useful for positioning a prompt, in this case, I want the contents of the matches, and s-match returns each grouping.

(defun ha-hamacs-edit--process-entry (rg-input)
  "Return list of heading, file and line number.
Parses the line entry, RG-INPUT, from a call to `rg',
using the regular expression, `ha-hamacs-edit-rx-ripgrep'.
Returns something like:

  (\"Some Heading\" \"some-file.org\" 42)"
  (seq-let (_ file line level head)
      (s-match ha-hamacs-edit-rx-ripgrep rg-input)
    (list (ha-hamacs-edit--new-heading file head (length level))
          file
          (string-to-number line))))

Before we dive into the implementation of this function, let’s write a test to validate (and explain) what we expect to return:

(ert-deftest ha-hamacs-edit--process-entry-test ()
  (setq ha-hamacs-edit-prev-head-list '())
  (should (equal
           (ha-hamacs-edit--process-entry
            "ha-somefile.org:42:* A Nice Headline  :ignored:")
           '("Somefile∷ A Nice Headline  " "ha-somefile.org" 42)))

  ;; For second-level headlines, we need to keep track of its parent,
  ;; and for this, we use a global variable, which we can set for the
  ;; purposes of this test:
  (setq ha-hamacs-edit-prev-head-list '("Parent"))
  (should (equal
           (ha-hamacs-edit--process-entry
            "ha-somefile.org:73:** Another Headline")
           '("Somefile∷ Parent﹥ Another Headline"
             "ha-somefile.org" 73)))

  (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"))
  (should (equal
           (ha-hamacs-edit--process-entry
            "ha-somefile.org:73:*** Deep Heading")
           '("Somefile∷ Parent﹥ Subparent﹥ Deep Heading"
             "ha-somefile.org" 73)))

  (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"
                                        "Subby" "Deepsubby"))
  (should (equal
           (ha-hamacs-edit--process-entry
            "ha-somefile.org:73:***** Deepest Heading")
           '("Somefile∷ ... Deepest Heading"
             "ha-somefile.org" 73))))

We next need a regular expression to pass to s-match to parse the output:

(defvar ha-hamacs-edit-rx-ripgrep
  (rx (group (one-or-more (not ":"))) ":"   ; filename
      (group (one-or-more digit)) ":"       ; line number
      (group (one-or-more "*"))             ; header asterisks
      (one-or-more space)
      (group (one-or-more (not ":"))))      ; headline without tags
  "Regular expression of ripgrep default output with groups.")

The —new-heading function will prepend the name of the file and its parent headlines (if any) to the headline to be more useful in both understanding the relative context of the headline, as well as better to search using fuzzy matching.

This context is especially important as completing-read will place the most recent choices at the top.

I found the use of setf to be quite helpful in manipulating the list of parents. Remember a list in a Lisp, is a linked list, and we can easily replace one or more parts, by pointing to an new list. This is my first iteration of this function, and I might come back and simplify it.

Essentially, if we get to a top-level headline, we set the list of parents to a list containing that new headline. If we get a second-level headine, B, and our parent list is A, we create a list ’(A B) by setting the cdr of ’(A) to the list ’(B). The advantage of this approach is that if the parent list is ’(A C D), the setf works the same, and the dangled sublist, ’(C D) gets garbage collected.

(defun ha-hamacs-edit--new-heading (file head level)
  "Return readable entry from FILE and org headline, HEAD.
The HEAD headline is, when LEVEL is greater than 1,
to include parent headlines. This is done by storing
the list of parents in `ha-hamacs-edit-prev-head-list'."
  ;; Reset the parent list to include the new HEAD:
  (cond
    ((= level 1)
     (setq ha-hamacs-edit-prev-head-list (list head)))
    ((= level 2)
     (setf (cdr ha-hamacs-edit-prev-head-list) (list head)))
    ((= level 3)
     (setf (cddr ha-hamacs-edit-prev-head-list) (list head)))
    ((= level 4)
     (setf (cdddr ha-hamacs-edit-prev-head-list) (list head)))
    ((= level 5)
     (setf (cddddr ha-hamacs-edit-prev-head-list) (list head))))
    ;; Let's never go any deeper than this...

  (format "%s∷ %s"
   (ha-hamacs-edit--file-title file)
   (s-join "﹥ " ha-hamacs-edit-prev-head-list)))

The following test should pass some mustard and explain how this function works:

(ert-deftest ha-hamacs-edit--new-heading-test ()
  (should (equal
           (ha-hamacs-edit--new-heading "ha-foobar.org" "Apples" 1)
           "Foobar∷ Apples"))
  (setq ha-hamacs-edit-prev-head-list '("Apples"))
  (should (equal
           (ha-hamacs-edit--new-heading "ha-foobar.org" "Oranges" 2)
           "Foobar∷ Apples﹥ Oranges"))
  (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges"))
  (should (equal
           (ha-hamacs-edit--new-heading "ha-foobar.org" "Bananas" 3)
           "Foobar∷ Apples﹥ Oranges﹥ Bananas"))
  (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges" "Bananas"))
  (should (equal
           (ha-hamacs-edit--new-heading "ha-foobar.org" "Cantaloupe" 4)
           "Foobar∷ Apples﹥ Oranges﹥ Bananas﹥ Cantaloupe")))

I store the current list of parents, in the following (gasp) global variable:

(defvar ha-hamacs-edit-prev-head-list '("" "")
  "The current parents of headlines as a list.")

I would like to make the filename more readable, I use the s-match again, to get the groups of a regular expression, remove all the dashes, and use s-titleize to capitalize each word:

(defun ha-hamacs-edit--file-title (file)
  "Return a more readable string from FILE."
  (s-with file
    (s-match ha-hamacs-edit-file-to-title)
    (second)
    (s-replace "-" " ")
    (s-titleize)))

(defvar ha-hamacs-edit-file-to-title
  (rx (optional (or "README-" "ha-"))
      (group (one-or-more any)) ".org")
  "Extract the part of a file to use as a title.")

So the following tests should pass:

(ert-deftest ha-hamacs-edit-file-title-test ()
  (should (equal (ha-hamacs-edit-file-title "ha-apples.org") "Apples"))
  (should (equal (ha-hamacs-edit-file-title "apples.org") "Apples"))
  (should (equal (ha-hamacs-edit-file-title "README-apples.org") "Apples"))
  (should (equal (ha-hamacs-edit-file-title "README.org") "Readme")))

Keybindings

With a lovely collection of functions, we need to have a way to easily call them. I’ve been using the pretty-hydra feature of major-mode-hydra:

(defvar org-babel--title (concat (all-the-icons-faicon "pencil-square-o")
                                 "  Literate Programming Support"))
(pretty-hydra-define org-babel
  (:title org-babel--title :color blue)
  ("Code Blocks"
   (("g" avy-jump-org-block "Goto ")
    ("j" org-next-block "Previous" :color pink)
    ("k" org-previous-block "Next" :color pink))
   "Evaluate"
   (("o" avy-org-babel-execute-src-block "Block ")
    ("h" org-babel-execute-subtree "Section")
    ("b" org-babel-execute-buffer "Buffer"))
   "Tangle"
   (("t" org-babel-tangle "to Default")
    ("f" org-babel-tangle-file "choose File")
    ("T" org-babel-detangle "from File"))
   "Misc"
   (("e" avy-org-babel-edit-src-block "Edit Block ")
    ("s" org-babel-pop-to-session-maybe "Session REPL")
    ("v" ha-org-babel-tangle-visit-file "Visit Tangled"))))

And tie this hydra into the existing leader system:

(ha-leader :keymaps 'org-mode-map "o s" '("babel" . org-babel/body))