I do a lot of /literate programming/ using capabilities found in the Org project. Over the years, I’ve smoothed some of the rough edges by writing supporting functions, collected below.
What are the advantage of [[https://en.wikipedia.org/wiki/Literate_programming][literate programming]]? I listed some in my essay and video about my [[https://howardism.org/Technical/Emacs/literate-devops.html][literate devops]] ideas, but a brief recap:
- ambiguous and/or complicated ideas can be reasoned about in prose before code
- links to ideas, code snippets and supporting projects are more accessible than in comments
- diagrams written in [[file:ha-org.org::*PlantUML][PlantUML]], [[file:ha-org.org::*Graphviz][Graphviz]], or [[file:ha-org.org::*Pikchr][Pikchr]] maintained and displayed with code
- full access to org features, like clocks, task lists and agendas
- mixing languages in the same file, e.g. shell instructions for installing supporting modules before the code that uses that module
If literate programming is so great, why doesn’t everyone do it? Most of the /advantages/ listed above are advantages only because we are using Emacs and Org Mode, and getting teams to adopt this requires changing editors and workflows. Therefore, literate programming is a solo endeavor.
Since we are using Emacs, the downsides of using a literate approach (even for personal projects) can be minimized.
*Note:* What follows is advanced LP usage. I would recommend checking out my [[https://www.howardism.org/Technical/Emacs/literate-programming-tutorial.html][Introduction to LP in Org]] before venturing down this essay.
After tangling a file, you might want to take a look at it. But a single org file can tangle out to more than one file, so how to choose? Each block could have its one file, as would the particular language-specific blocks in a section or the entire file. I don’t care to be too pendantic here, because I often know the name of the file.
Perhaps a little function to map a /language/ to a file name:
#+BEGIN_SRC emacs-lisp
(defun ha-org-babel-tangle-default (language)
"Return the default tangled file based on LANGUAGE."
(format "%s.%s"
(file-name-sans-extension (buffer-file-name))
(pcase language
("emacs-lisp" "el")
("elisp" "el")
("python" "py")
("ruby" "rb")
;; ...
("bash" "sh")
("sh" "sh")
("shell" "sh"))))
#+END_SRC
This function returns the name of the tangled file based on either the block or a default name based on the file name:
#+BEGIN_SRC emacs-lisp
(defun ha-org-babel-tangle-file-name ()
"Return _potential_ file name or Org source.
If nearest block has a `:tangle' entry, return that.
If block doesn't have `:tangle' (or set to `yes'),
return the filename based on the block's language."
(message "Looking for %s, which doesn't exist." tangled-file))))
#+END_SRC
*Note:* You can /return/ to the original Org file with =org-babel-tangle-jump-to-org=, if you set the tangle =comments= to =link=, as in this global property:
#+begin_example
,#+PROPERTY: header-args:emacs-lisp :tangle yes :comments link
I’ve been using Oleh Krehel’s (abo-abo) [[https://github.com/abo-abo/avy][Avy project]] to jump around the screen for years, and I just learned that I can wrap the =avy-jump= function to provide either/or regular expression and action to perform.
For instance, the following function can be used to quickly select a source code block, and jump to it:
At times I would like to jump to a particular block, evaluate the code, and jump back. This seems like a great job for the [[https://github.com/abo-abo/avy][avy project]]. The =avy-jump= function takes a regular expression of text /in the frame/ (which means you can specify text in other windows), and highlights each match. Normally, selecting a match moves the cursor to that match, the =avy-jump= accepts a function to execute instead:
In this case, =avy-org-babel-execute-src-block= highlights all /visible blocks/ on the frame, with a letter on each. Selecting the letter, evaluates that block without moving the cursor.
A trick to =org-babel-tangle=, is that it tangles /what Emacs shows/, that is, it tangles /visible/ code blocks after narrowing to the current org section. This means, we can call =org-narrow-to-subtree= to temporary hide everything in the org file except the current heading, evaluate all blocks in the “now visible” buffer, and then widen:
The Emacs interface for jumping to function definitions and variable declarations is called xref (see [[https://www.ackerleytng.com/posts/emacs-xref/][this great article]] for an overview of the interface). I think it would be great to be able, even within the prose of an org file, to jump to the definition of a function that is defined in an org file.
- [[*Definitions][Definitions]] :: To jump to the line where a macro, function or variable is defined.
- [[*References][References]] :: To get a list of all /calls/ or usage of a symbol, but only within code blocks.
- [[*Apropos][Apropos]] :: To get a list of all references, even within org-mode prose.
In a normal source code file, you know the language, so you have way of figuring out what a symbol is and how it could be defined in that language. In org files, however, one can use multiple languages, even in the same file.
In the code that follows, I’ve made an assumption that I will primarily use this xref interface for Emacs Lisp code, however, it wouldn’t take much (a single regular expression) to convert to another language.
Taking a cue from [[https://github.com/jacktasia/dumb-jump][dumb-jump]], I’ve decided to not attempt to build any sort of [[https://github.com/dedi/gxref/][tag interaction]], but instead, call [[https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md][ripgrep]]. I love that its =–-json= option outputs much more parseable text.
*** Symbols
I wrote the =ha-literate-symbol-at-point= function as an attempt at being clever with figuring out what sort of symbol references we would want from an org file. I assume that a symbol may be written surrounded by =~= or ~=~ characters (for code and verbatim text), as well as in quotes or braces, etc.
While the goal is Emacs Lisp (and it mostly works for that), it will probably work for other languages as well.
#+begin_src emacs-lisp
(defun ha-literate-symbol-at-point ()
"Return an alphanumeric sequence at point.
Assuming the sequence can be surrounded by typical
This helper function does the work of calling =ripgrep=, parsing its output, and filtering only the /matches/ line. Yes, an interesting feature of =rg= is that it spits out a /sequence/ of JSON-formatted text, so we can use =seq-filter= to grab lines that represent a match, and =seq-map= to “do the work”. Since we have a couple of ways of /doing the work/, we pass in a function, =processor=, which, along with transforming the results, could spit out =nulls=, so the =seq-filter= with the =identity= function eliminates that.
Since our =ripgrep= searches from the /project root/, but xref wants to make file references relative to the buffer that is calling it, we need to make some changes:
As mentioned above, let’s assume we can use =ripgrep= to search for /definitions/ in Lisp. I choose that because most of my literate programming is in Emacs Lisp. This regular expression should work with things like =defun= and =defvar=, etc. as well as =use-package=, allowing me to search for the /definition/ of an Emacs package:
The work of processing a match for the =ha-literate-definition= function. It calls =xref-make= to create an object for the Xref system. This takes two parameters, the text and the location. We create a location with =xref-make-file-location=.
While traditionally, =-apropos= can reference symbols in comments and documentation, searching for /references/ tend to be /calls/ and whatnot. What does that mean in the context of an org file? I’ve decided that references should only show symbols /within org blocks/.
How do we know we are /inside/ an org block?
I call =ripgrep= twice, once to get all the =begin_= and =end_src= lines and their line numbers.
The second =ripgrep= call gets the references.
#+begin_src emacs-lisp
(defun ha-literate-references (symb)
"Return list of `xref' objects for SYMB location in org files.
The location is limited only references in org blocks."
;; First, get and store the block line numbers:
(ha-literate--block-line-numbers)
;; Second, call `rg' again to get all matches of SYMB:
Notice for this function, we need a new processor that limits the results to only matches between the beginning and ending of a block, which I’ll describe later.
The =ha-literate--block-line-numbers= returns a hash where the keys are files, and the value is a series of begin/end line numbers. It calls =ripgrep=, but has a new processor.
#+begin_src emacs-lisp
(defun ha-literate--block-line-numbers ()
"Call `ripgrep' for org blocks and store results in a hash table.
And the function to process the output simply attempts to connect the =begin_src= with the =end_src= lines. In true Emacs Lisp fashion (where we can’t easily, lexically nest functions), we use a global variable:
With a collection of line numbers for all org-blocks in all org files in our project, we can process a particular match from =ripgrep= to see if the match is /within/ a block. Since the key is a file, and =.data.path.text= is the filename, that part is done, but we need a helper to walk down the list.
The helper function, =ha-literate--process-in-block= is a /recursive/ function that takes each tuple and sees if =line-number= is between them. If it isn’t between any tuple, and the list is empty, then we return =nil= to filter that out later.
At this point, we can jump to functions and variables that I define in my org file, or even references to standard symbols like =xref-make= or =xref-backend-functions=.
I can jump around my literate code as if they were =.el= files. I may want to think about expanding the definitions to figure out the language of the destination.
As large literate programming projects grow, I refine, re-organize and refactor content. I don’t always remember where I put particular code. For instance, in my Emacs configuration, did I configure /eww/, in [[file:ha-config.org][my default config]] file, or did I move it somewhere? Originally, after loading the file, I could issue a call to [[file:ha-general.org::*Consult][consult-imenu]] to get to the right location, but that assumes I have the correct file loaded.
The following section shows some code to use the fuzzy matching features of [[file:ha-config.org::*Orderless][Orderless]], to choose a headline in any of my Org files in a project, and then load that file and jump to that headline. The interface is =ha-hamacs-edit-file-heading=, and the supporting functions begin with =ha-hamacs-edit-=:
This function collects all possible headers by issuing a call to =ripgrep=, which returns something like:
#+begin_example
ha-applications.org:29:* Git and Magit
ha-applications.org:85:** Git Gutter
ha-applications.org:110:** Git Delta
ha-applications.org:136:** Git with Difftastic
...
"ha-applications.org:385:* Web Browsing
ha-applications.org:386:** EWW
...
#+end_example
We then filter out non-useful headers (with =ha-hamcs-edit—filter-heading=), and convert the headlines with =ha-hamcs-edit—process-entry= to be more presentable:
As the above function’s documentation string claims, I create a list that contains the data structure necessary for =completing-read= as well as the information I need to load/jump to a position in the file. This is a three-element list of the /headline/, /filename/ and /line number/ for each entry:
#+begin_src emacs-lisp :tangle no
'(("Applications∷ Git and Magit" "ha-applications.org" 29)
("Applications∷ Git and Magit ﹥ Git Gutter" "ha-applications.org" 85)
("Applications∷ Git and Magit ﹥ Git Delta" "ha-applications.org" 110)
("Applications∷ Git and Magit ﹥ Time Machine" "ha-applications.org" 265)
("Applications∷ Git and Magit ﹥ Gist" "ha-applications.org" 272)
("Applications∷ Git and Magit ﹥ Forge" "ha-applications.org" 296)
("Applications∷ Git and Magit ﹥ Pushing is Bad" "ha-applications.org" 334)
("Applications∷ Git and Magit ﹥ Github Search?" "ha-applications.org" 347)
("Applications∷ ediff" "ha-applications.org" 360)
("Applications∷ Web Browsing" "ha-applications.org" 385)
("Applications∷ Web Browsing ﹥ EWW" "ha-applications.org" 386)
;; ...
)
#+end_src
We’ll use this shell command to call =ripgrep= to search my collection of org files:
#+begin_src emacs-lisp
(defvar ha-hamacs-edit-ripgrep-headers
(concat "rg"
" --no-heading"
" --line-number"
;; " --max-depth 1"
" -e '^\\*+ '"
" *.org")
"A ripgrep shell call to search my headers.")
#+end_src
Not every header should be a destination, as many of my org files have duplicate headlines, like *Introduction* and *Technical Artifacts*, so I can create a regular expression to remove or flush entries:
And this next function is callable by the filter function, it uses the regular expression and returns true (well, non-nil) if the line entry given, =rg-input=, should be removed:
#+begin_src emacs-lisp
(defun ha-hamacs-edit--filter-heading (rg-input)
"Return non-nil if we should remove RG-INPUT.
These are headings with typical, non-unique entries,
The =seq-map= needs to take each line from the =ripgrep= call and convert it to a list that I can use for the =completing-read= prompt. I love the combination of =seq-let= and =s-match= from Magnar’s [[https://github.com/magnars/s.el][String library]]. The built-in function, =string-match= returns the index in the string where the match occurs, and this is useful for positioning a prompt, in this case, I want the /contents/ of the matches, and =s-match= returns each /grouping/.
#+begin_src emacs-lisp
(defun ha-hamacs-edit--process-entry (rg-input)
"Return list of heading, file and line number.
Parses the line entry, RG-INPUT, from a call to `rg',
using the regular expression, `ha-hamacs-edit-rx-ripgrep'.
Returns something like:
(\"Some Heading\" \"some-file.org\" 42)"
(seq-let (_ file line level head)
(s-match ha-hamacs-edit-rx-ripgrep rg-input)
(list (ha-hamacs-edit--new-heading file head (length level))
file
(string-to-number line))))
#+end_src
Before we dive into the implementation of this function, let’s write a test to validate (and explain) what we expect to return:
(group (one-or-more (not ":")))) ; headline without tags
"Regular expression of ripgrep default output with groups.")
#+end_src
The =—new-heading= function will /prepend/ the name of the file and its parent headlines (if any) to the headline to be more useful in both understanding the relative context of the headline, as well as better to search using fuzzy matching.
This /context/ is especially important as =completing-read= will place the most recent choices at the top.
I found the use of =setf= to be quite helpful in manipulating the list of parents. Remember a =list= in a Lisp, is a /linked list/, and we can easily replace one or more parts, by pointing to an new list. This is my first iteration of this function, and I might come back and simplify it.
Essentially, if we get to a top-level headline, we set the list of parents to a list containing that new headline. If we get a second-level headine, =B=, and our parent list is =A=, we create a list =’(A B)= by setting the =cdr= of =’(A)= to the list =’(B)=. The advantage of this approach is that if the parent list is =’(A C D)=, the =setf= works the same, and the dangled /sublist/, =’(C D)= gets garbage collected.
#+begin_src emacs-lisp
(defun ha-hamacs-edit--new-heading (file head level)
"Return readable entry from FILE and org headline, HEAD.
The HEAD headline is, when LEVEL is greater than 1,
to include parent headlines. This is done by storing
the list of parents in `ha-hamacs-edit-prev-head-list'."
I store the current list of parents, in the following (/gasp/) /global variable/:
#+begin_src emacs-lisp
(defvar ha-hamacs-edit-prev-head-list '("" "")
"The current parents of headlines as a list.")
#+end_src
I would like to make the /filename/ more readable, I use the =s-match= again, to get the groups of a regular expression, remove all the dashes, and use =s-titleize= to capitalize each word:
With a lovely collection of functions, we need to have a way to easily call them. I’ve been using the =pretty-hydra= feature of [[https://github.com/jerrypnz/major-mode-hydra.el][major-mode-hydra]]:
