21 KiB
Literate Programming with Org
A literate programming file for literate programming in Emacs Org Files.
Introduction
I do a lot of literate programming using capabilities found in the Org project. Over the years, I’ve smoothed some of the rough edges by writing supporting functions, collected below.
What are the advantage of literate programming? I listed some in my essay and video about my literate devops ideas, but a brief recap:
- ambiguous and/or complicated ideas can be reasoned about in prose before code
- links to ideas, code snippets and supporting projects are more accessible than in comments
- diagrams written in PlantUML, Graphviz, or Pikchr maintained and displayed with code
- full access to org features, like clocks, task lists and agendas
- mixing languages in the same file, e.g. shell instructions for installing supporting modules before the code that uses that module
If literate programming is so great, why doesn’t everyone do it? Most of the advantages listed above are advantages only because we are using Emacs and Org Mode, and getting teams to adopt this requires changing editors and workflows. Therefore, literate programming is a solo endeavor.
Since we are using Emacs, the downsides of using a literate approach (even for personal projects) can be minimized.
Note: What follows is advanced LP usage. I would recommend checking out my Introduction to LP in Org before venturing down this essay.
Navigating Code Blocks
I’ve been using Oleh Krehel’s (abo-abo) Avy project to jump around the screen for years, and I just learned that I can wrap the avy-jump function to provide either/or regular expression and action to perform.
For instance, the following function can be used to quickly select a source code block, and jump to it:
(defun avy-jump-org-block ()
"Jump to org block using Avy subsystem."
(interactive)
(avy-jump (rx "#+begin_src ") :action 'goto-char))I need to take advantage of this feature more.
Evaluating Code
Hitting C-c C-c in a source code block evaluates the code. Simple, sure, but the following enhancements make this more accessible.
Evaluating a Block
At times I would like to jump to a particular block, evaluate the code, and jump back. This seems like a great job for the avy project. The avy-jump function takes a regular expression of text in the frame (which means you can specify text in other windows), and highlights each match. Normally, selecting a match moves the cursor to that match, the avy-jump accepts a function to execute instead:
(defun org-babel-execute-src-block-at-point (&optional point)
"Call `org-babel-execute-src-block' at POINT."
(save-excursion
(goto-char point)
(org-babel-execute-src-block)))
(defun avy-org-babel-execute-src-block ()
"Call `org-babel-execute-src-block' on block given by Avy.
Use Avy subsystem to select a visible Org source code block,
e.g. `#+begin_src', and then executes the code without moving
the point."
(interactive)
(avy-jump (rx "#+begin_src ")
:action 'org-babel-execute-src-block-at-point))
In this case, avy-org-babel-execute-src-block highlights all visible blocks on the frame, with a letter on each. Selecting the letter, evaluates that block without moving the cursor.
TODO Screenshot of multiple highlighted blocks.
Evaluating a Section
A trick to org-babel-tangle, is that it tangles what is shown, that is, it will only tangle code blocks that are visible after narrowing to the current org section. This means, we can call org-narrow-to-subtree to temporary hide everything in the org file except the current heading, evaluate all blocks in the “now visible” buffer, and then widen:
(defun org-babel-execute-subtree ()
"Execute all Org source blocks in current subtree."
(interactive "P")
(save-excursion
(org-narrow-to-subtree)
(org-babel-execute-buffer)
(widen)))Editing a Block
Why navigate to a block, just to focus on that block in a dedicated buffer, when we can take advantage of the avy-jump and edit any visible block?
(defun org-babel-edit-src-block-at-point (&optional point)
"Call `org-babel-execute-src-block' at POINT."
(save-excursion
(goto-char point)
(org-edit-src-code)))
(defun avy-org-babel-edit-src-block ()
"Call `org-edit-src-code' on block given by Avy.
Use Avy subsystem to select a visible Org source code block,
e.g. `#+begin_src', and then executes the code without moving
the point."
(interactive)
(avy-jump (rx "#+begin_src ")
:action
'org-babel-edit-src-block-at-point))Finding Code
One of the issues with literate programming is not being able to use the same interface for moving around code when the source code is in org files.
Searching by Function Name
I wrote a function, ha-org-code-block-jump to use the standard xref interface to jump to a function definition in the literate org file. Since the code is specific to Emacs Lisp (the bulk of my literate programming code is in Lisp), I’m leaving it in my programming-elisp configuration.
TODO: Do all the xref- functions for search an collection of org files, not just definition.
Searching by Header
As large literate programming projects grow, I refine, re-organize and refactor content. I don’t always remember where I put particular code. For instance, in my Emacs configuration, did I configure eww, in my default config file, or did I move it somewhere? Originally, after loading the file, I could issue a call to consult-imenu to get to the right location, but that assumes I have the correct file loaded.
The following section shows some code to use the fuzzy matching features of Orderless, to choose a headline in any of my Org files in a project, and then load that file and jump to that headline. The interface is ha-hamacs-edit-file-heading, and the supporting functions begin with ha-hamacs-edit-:
(defun ha-hamacs-edit-file-heading (&optional project-root)
"Edit a file based on a particular heading.
After presenting list of headings from all Org files,
it loads the file, and jumps to the line number where
the heading is located."
(interactive)
(let* ((default-directory (or project-root (project-root (project-current))))
(file-headings (ha-hamacs-edit--file-heading-list project-root))
(file-choice (completing-read "Edit Heading: " file-headings))
(file-tuple (alist-get file-choice file-headings
nil nil 'string-equal)))
(find-file (first file-tuple))
(goto-line (second file-tuple))))
This function collects all possible headers by issuing a call to ripgrep, which returns something like:
ha-applications.org:29:* Git and Magit ha-applications.org:85:** Git Gutter ha-applications.org:110:** Git Delta ha-applications.org:136:** Git with Difftastic ... "ha-applications.org:385:* Web Browsing ha-applications.org:386:** EWW ...
We then filter out non-useful headers (with ha-hamcs-edit—filter-heading), and convert the headlines with ha-hamcs-edit—process-entry to be more presentable:
(defun ha-hamacs-edit--file-heading-list (&optional project-root)
"Return list of lists of headlines and file locations.
Using the output from the shell command, `ha-hamacs-edit-ripgrep-headers',
it parses and returns something like:
'((\"Applications∷ Git and Magit\" \"ha-applications.org\" 29)
(\"Applications∷ Git and Magit ﹥ Git Gutter\" \"ha-applications.org\" 85)
(\"Applications∷ Git and Magit ﹥ Git Delta\" \"ha-applications.org\" 110)
(\"Applications∷ Git and Magit ﹥ Time Machine\" \"ha-applications.org\" 265)
...)"
(let ((default-directory (or project-root (project-root (project-current)))))
(thread-last ha-hamacs-edit-ripgrep-headers
(shell-command-to-list)
;; Let's remove non-helpful, duplicate headings,
;; like Introduction:
(seq-remove 'ha-hamacs-edit--filter-heading)
;; Convert the results into both a displayable
;; string as well as the file and line structure:
(seq-map 'ha-hamacs-edit--process-entry))))
As the above function’s documentation string claims, I create a list that contains the data structure necessary for completing-read as well as the information I need to load/jump to a position in the file. This is a three-element list of the headline, filename and line number for each entry:
'(("Applications∷ Git and Magit" "ha-applications.org" 29)
("Applications∷ Git and Magit ﹥ Git Gutter" "ha-applications.org" 85)
("Applications∷ Git and Magit ﹥ Git Delta" "ha-applications.org" 110)
("Applications∷ Git and Magit ﹥ Time Machine" "ha-applications.org" 265)
("Applications∷ Git and Magit ﹥ Gist" "ha-applications.org" 272)
("Applications∷ Git and Magit ﹥ Forge" "ha-applications.org" 296)
("Applications∷ Git and Magit ﹥ Pushing is Bad" "ha-applications.org" 334)
("Applications∷ Git and Magit ﹥ Github Search?" "ha-applications.org" 347)
("Applications∷ ediff" "ha-applications.org" 360)
("Applications∷ Web Browsing" "ha-applications.org" 385)
("Applications∷ Web Browsing ﹥ EWW" "ha-applications.org" 386)
;; ...
)
We’ll use this shell command to call ripgrep to search my collection of org files:
(defvar ha-hamacs-edit-ripgrep-headers
(concat "rg"
" --no-heading"
" --line-number"
;; " --max-depth 1"
" -e '^\\*+ '"
" *.org")
"A ripgrep shell call to search my headers.")Not every header should be a destination, as many of my org files have duplicate headlines, like Introduction and Technical Artifacts, so I can create a regular expression to remove or flush entries:
(defvar ha-hamacs-edit-flush-headers
(rx "*" (one-or-more space)
(or "Introduction"
"Install"
"Overview"
"Summary"
"Technical Artifacts"))
"Regular expression matching headers to purge.")
Note: This variable should be set in the .dir-locals.el for a particular project, as in:
((org-mode . ((ha-hamacs-edit-flush-headers .
"\\*[[:space:]]+\\(?:Background\\|Summary\\)"))))
And this next function is callable by the filter function, it uses the regular expression and returns true (well, non-nil) if the line entry given, rg-input, should be removed:
(defun ha-hamacs-edit--filter-heading (rg-input)
"Return non-nil if we should remove RG-INPUT.
These are headings with typical, non-unique entries,
like Introduction and Summary."
(string-match ha-hamacs-edit-flush-headers rg-input))
The seq-map needs to take each line from the ripgrep call and convert it to a list that I can use for the completing-read prompt. I love the combination of seq-let and s-match from Magnar’s String library. The built-in function, string-match returns the index in the string where the match occurs, and this is useful for positioning a prompt, in this case, I want the contents of the matches, and s-match returns each grouping.
(defun ha-hamacs-edit--process-entry (rg-input)
"Return list of heading, file and line number.
Parses the line entry, RG-INPUT, from a call to `rg',
using the regular expression, `ha-hamacs-edit-rx-ripgrep'.
Returns something like:
(\"Some Heading\" \"some-file.org\" 42)"
(seq-let (_ file line level head)
(s-match ha-hamacs-edit-rx-ripgrep rg-input)
(list (ha-hamacs-edit--new-heading file head (length level))
file
(string-to-number line))))Before we dive into the implementation of this function, let’s write a test to validate (and explain) what we expect to return:
(ert-deftest ha-hamacs-edit--process-entry-test ()
(setq ha-hamacs-edit-prev-head-list '())
(should (equal
(ha-hamacs-edit--process-entry
"ha-somefile.org:42:* A Nice Headline :ignored:")
'("Somefile∷ A Nice Headline " "ha-somefile.org" 42)))
;; For second-level headlines, we need to keep track of its parent,
;; and for this, we use a global variable, which we can set for the
;; purposes of this test:
(setq ha-hamacs-edit-prev-head-list '("Parent"))
(should (equal
(ha-hamacs-edit--process-entry
"ha-somefile.org:73:** Another Headline")
'("Somefile∷ Parent﹥ Another Headline"
"ha-somefile.org" 73)))
(setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"))
(should (equal
(ha-hamacs-edit--process-entry
"ha-somefile.org:73:*** Deep Heading")
'("Somefile∷ Parent﹥ Subparent﹥ Deep Heading"
"ha-somefile.org" 73)))
(setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"
"Subby" "Deepsubby"))
(should (equal
(ha-hamacs-edit--process-entry
"ha-somefile.org:73:***** Deepest Heading")
'("Somefile∷ ... Deepest Heading"
"ha-somefile.org" 73))))
We next need a regular expression to pass to s-match to parse the output:
(defvar ha-hamacs-edit-rx-ripgrep
(rx (group (one-or-more (not ":"))) ":" ; filename
(group (one-or-more digit)) ":" ; line number
(group (one-or-more "*")) ; header asterisks
(one-or-more space)
(group (one-or-more (not ":")))) ; headline without tags
"Regular expression of ripgrep default output with groups.")
The —new-heading function will prepend the name of the file and its parent headlines (if any) to the headline to be more useful in both understanding the relative context of the headline, as well as better to search using fuzzy matching.
This context is especially important as completing-read will place the most recent choices at the top.
I found the use of setf to be quite helpful in manipulating the list of parents. Remember a list in a Lisp, is a linked list, and we can easily replace one or more parts, by pointing to an new list. This is my first iteration of this function, and I might come back and simplify it.
Essentially, if we get to a top-level headline, we set the list of parents to a list containing that new headline. If we get a second-level headine, B, and our parent list is A, we create a list ’(A B) by setting the cdr of ’(A) to the list ’(B). The advantage of this approach is that if the parent list is ’(A C D), the setf works the same, and the dangled sublist, ’(C D) gets garbage collected.
(defun ha-hamacs-edit--new-heading (file head level)
"Return readable entry from FILE and org headline, HEAD.
The HEAD headline is, when LEVEL is greater than 1,
to include parent headlines. This is done by storing
the list of parents in `ha-hamacs-edit-prev-head-list'."
;; Reset the parent list to include the new HEAD:
(cond
((= level 1)
(setq ha-hamacs-edit-prev-head-list (list head)))
((= level 2)
(setf (cdr ha-hamacs-edit-prev-head-list) (list head)))
((= level 3)
(setf (cddr ha-hamacs-edit-prev-head-list) (list head)))
((= level 4)
(setf (cdddr ha-hamacs-edit-prev-head-list) (list head)))
((= level 5)
(setf (cddddr ha-hamacs-edit-prev-head-list) (list head))))
;; Let's never go any deeper than this...
(format "%s∷ %s"
(ha-hamacs-edit--file-title file)
(s-join "﹥ " ha-hamacs-edit-prev-head-list)))The following test should pass some mustard and explain how this function works:
(ert-deftest ha-hamacs-edit--new-heading-test ()
(should (equal
(ha-hamacs-edit--new-heading "ha-foobar.org" "Apples" 1)
"Foobar∷ Apples"))
(setq ha-hamacs-edit-prev-head-list '("Apples"))
(should (equal
(ha-hamacs-edit--new-heading "ha-foobar.org" "Oranges" 2)
"Foobar∷ Apples﹥ Oranges"))
(setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges"))
(should (equal
(ha-hamacs-edit--new-heading "ha-foobar.org" "Bananas" 3)
"Foobar∷ Apples﹥ Oranges﹥ Bananas"))
(setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges" "Bananas"))
(should (equal
(ha-hamacs-edit--new-heading "ha-foobar.org" "Cantaloupe" 4)
"Foobar∷ Apples﹥ Oranges﹥ Bananas﹥ Cantaloupe")))I store the current list of parents, in the following (gasp) global variable:
(defvar ha-hamacs-edit-prev-head-list '("" "")
"The current parents of headlines as a list.")
I would like to make the filename more readable, I use the s-match again, to get the groups of a regular expression, remove all the dashes, and use s-titleize to capitalize each word:
(defun ha-hamacs-edit--file-title (file)
"Return a more readable string from FILE."
(s-with file
(s-match ha-hamacs-edit-file-to-title)
(second)
(s-replace "-" " ")
(s-titleize)))
(defvar ha-hamacs-edit-file-to-title
(rx (optional (or "README-" "ha-"))
(group (one-or-more any)) ".org")
"Regular expression for extracting the interesting part of a
file to use as a title.")So the following tests should pass:
(ert-deftest ha-hamacs-edit-file-title-test ()
(should (equal (ha-hamacs-edit-file-title "ha-apples.org") "Apples"))
(should (equal (ha-hamacs-edit-file-title "apples.org") "Apples"))
(should (equal (ha-hamacs-edit-file-title "README-apples.org") "Apples"))
(should (equal (ha-hamacs-edit-file-title "README.org") "Readme")))Keybindings
With a lovely collection of functions, we need to have a way to easily call them. I’ve been using the pretty-hydra feature of major-mode-hydra:
(pretty-hydra-define org-babel (:color blue)
("Navigate"
(("g" avy-jump-org-block "goto")
("j" org-next-block "previous block" :color pink)
("k" org-previous-block "next block" :color pink))
"Evaluate"
(("o" avy-org-babel-execute-src-block "block")
("h" org-babel-execute-subtree "section")
("b" org-babel-execute-buffer "buffer"))
"Tangle"
(("t" org-babel-tangle "to File")
("T" org-babel-detangle "from File"))
"Misc"
(("e" avy-org-babel-edit-src-block "edit"))))And tie this hydra into the existing leader system:
(ha-leader :keymaps 'org-mode-map "o s" '("babel" . org-babel/body))