hamacs/ha-org-literate.org

#+title:  Literate Programming with Org
#+author: Howard Abrams
#+date:   2024-07-07
#+filetags: emacs hamacs
#+lastmod: [2024-07-07 Sun]

A literate programming file for literate programming in Emacs Org Files.

#+begin_src emacs-lisp :exports none
  ;;; ha-org-literate --- literate programming helpers -*- lexical-binding: t; -*-
  ;;
  ;; © 2024 Howard Abrams
  ;;   Licensed under a Creative Commons Attribution 4.0 International License.
  ;;   See http://creativecommons.org/licenses/by/4.0/
  ;;
  ;; Author: Howard Abrams <http://gitlab.com/howardabrams>
  ;; Maintainer: Howard Abrams
  ;; Created: July  7, 2024
  ;;
  ;; While obvious, GNU Emacs does not include this file or project.
  ;;
  ;;; Commentary:
  ;;
  ;; This file contains a collection of functions to easy some of the
  ;; sharp edges when doing literate programming in Org files.
  ;;
  ;; *NB:* Do not edit this file. Instead, edit the original
  ;; literate file at:
  ;;            /home/howard/other/hamacs/ha-org-literate.org
  ;; And tangle the file to recreate this one.
  ;;
  ;;; Code:
#+end_src

* Introduction
I do a lot of /literate programming/ using capabilities found in the Org project. Over the years, I’ve smoothed some of the rough edges by writing supporting functions, collected below.

What are the advantage of [[https://en.wikipedia.org/wiki/Literate_programming][literate programming]]? I listed some in my essay and video about my [[https://howardism.org/Technical/Emacs/literate-devops.html][literate devops]] ideas, but a brief recap:

  - ambiguous and/or complicated ideas can be reasoned about in prose before code
  - links to ideas, code snippets and supporting projects are more accessible than in comments
  - diagrams written in [[file:ha-org.org::*PlantUML][PlantUML]], [[file:ha-org.org::*Graphviz][Graphviz]], or [[file:ha-org.org::*Pikchr][Pikchr]] maintained and displayed with code
  - full access to org features, like clocks, task lists and agendas
  - mixing languages in the same file, e.g. shell instructions for installing supporting modules before the code that uses that module

If literate programming is so great, why doesn’t everyone do it? Most of the /advantages/ listed above are advantages only because we are using Emacs and Org Mode, and getting teams to adopt this requires changing editors and workflows. Therefore, literate programming is a solo endeavor.

Since we are using Emacs, the downsides of using a literate approach (even for personal projects) can be minimized.

*Note:* What follows is advanced LP usage. I would recommend checking out my [[https://www.howardism.org/Technical/Emacs/literate-programming-tutorial.html][Introduction to LP in Org]] before venturing down this essay.
* Navigating Code Blocks
I’ve been using Oleh Krehel’s (abo-abo) [[https://github.com/abo-abo/avy][Avy project]] to jump around the screen for years, and I just learned that I can wrap the =avy-jump= function to provide either/or regular expression and action to perform.

For instance, the following function can be used to quickly select a source code block, and jump to it:

#+begin_src emacs-lisp
  (defun avy-jump-org-block ()
    "Jump to org block using Avy subsystem."
    (interactive)
    (avy-jump (rx "#+begin_src ") :action 'goto-char))
#+end_src

I need to take advantage of this feature more.
* Evaluating Code
Hitting ~C-c C-c~ in a source code block /evaluates/ the code. Simple, sure, but the following enhancements make this more accessible.
** Syncing Tangled Code Automatically
Any file can set [[https://emacsdocs.org/docs/emacs/File-Variables][file-local Emacs variables]] when a file is loaded, but we can also run Emacs functions with the =eval:= sequence. If you place the following code at the bottom of your Org file, saving the buffer automatically tangles it:

#+begin_src org
  # Local Variables:
  # eval: (add-hook 'after-save-hook #'org-babel-tangle t t)
  # End:
#+end_src

If you set the =comments= header argument to =link=, you can actually make direct changes to your tangled code to have them update your original literate org file. For instance, add:

#+begin_src org
#+PROPERTY:    header-args: :tangle yes :comments link
#+end_src

See also the [[https://gitlab.com/mtekman/org-tanglesync.el][org-tanglesync]] project for putting a bit of controls around this feature.

Another idea is to call =org-babel-execute-buffer= when a file is loaded to automatically evaluate all blocks:

#+begin_src org
# eval:       (org-babel-execute-buffer)
#+end_src

Personally, I find that I would like to call this function manually instead of automatically.
** Evaluating a Block
At times I would like to jump to a particular block, evaluate the code, and jump back. This seems like a great job for the [[https://github.com/abo-abo/avy][avy project]]. The =avy-jump= function takes a regular expression of text /in the frame/ (which means you can specify text in other windows), and highlights each match. Normally, selecting a match moves the cursor to that match, the =avy-jump= accepts a function to execute instead:

#+begin_src emacs-lisp
  (defun org-babel-execute-src-block-at-point (&optional point)
    "Call `org-babel-execute-src-block' at POINT."
    (save-excursion
      (goto-char point)
      (org-babel-execute-src-block)))

  (defun avy-org-babel-execute-src-block ()
    "Call `org-babel-execute-src-block' on block given by Avy.
  Use Avy subsystem to select a visible Org source code block,
  e.g. `#+begin_src', and then executes the code without moving
  the point."
    (interactive)
    (avy-jump (rx "#+begin_src ")
              :action
              'org-babel-execute-src-block-at-point))
#+end_src

** Evaluating a Section
A trick to =org-babel-tangle=, is that it tangles /what is available/, that is, it will only tangle code blocks that are visible after narrowing to the current org section.

#+begin_src emacs-lisp :results silent
  (defun org-babel-execute-subtree ()
    "Execute all Org source blocks in current subtree."
    (interactive "P")
    (save-excursion
      (org-narrow-to-subtree)
      (org-babel-execute-buffer)
      (widen)))
#+end_src
** Editing a Block
#+begin_src emacs-lisp
  (defun org-babel-edit-src-block-at-point (&optional point)
    "Call `org-babel-execute-src-block' at POINT."
    (save-excursion
      (goto-char point)
      (org-edit-src-code)))

  (defun avy-org-babel-edit-src-block ()
    "Call `org-edit-src-code' on block given by Avy.
  Use Avy subsystem to select a visible Org source code block,
  e.g. `#+begin_src', and then executes the code without moving
  the point."
    (interactive)
    (avy-jump (rx "#+begin_src ")
              :action
              'org-babel-edit-src-block-at-point))
#+end_src

* Finding Code
One of the issues with literate programming is using the same interface for moving around code when the source code is in org files.

** Searching by Function Name
I wrote a function, =ha-org-code-block-jump= to use the standard =xref= interface to jump to a function definition /in the literate org file/. Since the code is specific to /Emacs Lisp/ (the bulk of my literate programming code is in Lisp), I’m leaving it in my [[file:ha-programming-elisp.org::*Goto Definitions][programming-elisp]] configuration.

TODO: Do all the =xref-= functions for search an collection of org files, not just definition.
** Searching by Header
As large literate programming projects grow, I refine, re-organize and refactor content. I don’t always remember where I put particular code. For instance, in my Emacs configuration, did I configure /eww/, in [[file:ha-config.org][my default config]] file, or did I move it somewhere?  Originally, after loading the file, I  could issue a call to [[file:ha-general.org::*Consult][consult-imenu]] to get to the right location, but that assumes I have the correct file loaded.

The following section shows some code I wrote one evening, to use the fuzzy matching features of [[file:ha-config.org::*Orderless][Orderless]], to choose a headline in any of my Org files in a project, and then load that file and jump to that headline. The interface is =ha-hamacs-edit-file-heading=, and the supporting functions begin with =ha-hamacs-edit-=:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit-file-heading (&optional project-root)
    "Edit a file based on a particular heading.
  After presenting list of headings from all Org files,
  it loads the file, and jumps to the line number where
  the heading is located."
    (interactive)
    (let* ((default-directory (or project-root (project-root (project-current))))
           (file-headings (ha-hamacs-edit--file-heading-list project-root))
           (file-choice   (completing-read "Edit Heading: " file-headings))
           (file-tuple    (alist-get file-choice file-headings
                                     nil nil 'string-equal)))
      (find-file (first file-tuple))
      (goto-line (second file-tuple))))
#+end_src

This function  collects all possible headers by issuing a call to =ripgrep=, which returns something like:

#+begin_example
ha-applications.org:29:* Git and Magit
ha-applications.org:85:** Git Gutter
ha-applications.org:110:** Git Delta
ha-applications.org:136:** Git with Difftastic
...
"ha-applications.org:385:* Web Browsing
ha-applications.org:386:** EWW
...
#+end_example

We then filter out non-useful headers (with =ha-hamcs-edit—filter-heading=), and convert the headlines with =ha-hamcs-edit—process-entry= to be more presentable:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--file-heading-list (&optional project-root)
    "Return list of lists of headlines and file locations.
  Using the output from the shell command, `ha-hamacs-edit-ripgrep-headers',
  it parses and returns something like:

  '((\"Applications∷ Git and Magit\" \"ha-applications.org\" 29)
    (\"Applications∷ Git and Magit ﹥ Git Gutter\" \"ha-applications.org\" 85)
    (\"Applications∷ Git and Magit ﹥ Git Delta\" \"ha-applications.org\" 110)
    (\"Applications∷ Git and Magit ﹥ Time Machine\" \"ha-applications.org\" 265)
    ...)"
    (let ((default-directory (or project-root (project-root (project-current)))))
      (thread-last ha-hamacs-edit-ripgrep-headers
                   (shell-command-to-list)
                   ;; Let's remove non-helpful, duplicate headings,
                   ;; like Introduction:
                   (seq-remove 'ha-hamacs-edit--filter-heading)
                   ;; Convert the results into both a displayable
                   ;; string as well as the file and line structure:
                   (seq-map 'ha-hamacs-edit--process-entry))))
#+end_src

As the above function’s documentation string claims, I create a list that contains the data structure necessary for =completing-read= as well as the information I need to load/jump to a position in the file. This is a three-element list of the /headline/, /filename/ and /line number/ for each entry:

#+begin_src emacs-lisp :tangle no
  '(("Applications∷ Git and Magit" "ha-applications.org" 29)
    ("Applications∷ Git and Magit ﹥ Git Gutter" "ha-applications.org" 85)
    ("Applications∷ Git and Magit ﹥ Git Delta" "ha-applications.org" 110)
    ("Applications∷ Git and Magit ﹥ Time Machine" "ha-applications.org" 265)
    ("Applications∷ Git and Magit ﹥ Gist" "ha-applications.org" 272)
    ("Applications∷ Git and Magit ﹥ Forge" "ha-applications.org" 296)
    ("Applications∷ Git and Magit ﹥ Pushing is Bad" "ha-applications.org" 334)
    ("Applications∷ Git and Magit ﹥ Github Search?" "ha-applications.org" 347)
    ("Applications∷ ediff" "ha-applications.org" 360)
    ("Applications∷ Web Browsing" "ha-applications.org" 385)
    ("Applications∷ Web Browsing ﹥ EWW" "ha-applications.org" 386)
    ;; ...
    )
#+end_src

We’ll use this shell command to call  =ripgrep= to search my collection of org files:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-ripgrep-headers
    (concat "rg"
            " --no-heading"
            " --line-number"
            ;; " --max-depth 1"
            " -e '^\\*+ '"
            " *.org")
    "A ripgrep shell call to search my headers.")
#+end_src

Not every header should be a destination, as many of my org files have duplicate headlines, like *Introduction* and *Technical Artifacts*, so I can create a regular expression to remove or flush entries:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-flush-headers
    (rx "*" (one-or-more space)
        (or "Introduction"
            "Install"
            "Overview"
            "Summary"
            "Technical Artifacts"))
    "Regular expression matching headers to purge.")
#+end_src

And this next function is callable by the filter function, it uses the regular expression and returns true (well, non-nil) if the line entry given, =rg-input=, should be removed:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--filter-heading (rg-input)
    "Return non-nil if we should remove RG-INPUT.
  These are headings with typical, non-unique entries,
  like Introduction and Summary."
    (string-match ha-hamacs-edit-flush-headers rg-input))
#+end_src

The =seq-map= needs to take each line from the =ripgrep= call and convert it to a list that I can use for the =completing-read= prompt. I love the combination of =seq-let= and =s-match= from Magnar’s [[https://github.com/magnars/s.el][String library]]. The built-in function, =string-match= returns the index in the string where the match occurs, and this is useful for positioning a prompt, in this case, I want the /contents/ of the matches, and =s-match= returns each /grouping/.

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--process-entry (rg-input)
    "Return list of heading, file and line number.
  Parses the line entry, RG-INPUT, from a call to `rg',
  using the regular expression, `ha-hamacs-edit-rx-ripgrep'.
  Returns something like:

    (\"Some Heading\" \"some-file.org\" 42)"
    (seq-let (_ file line level head)
        (s-match ha-hamacs-edit-rx-ripgrep rg-input)
      (list (ha-hamacs-edit--new-heading file head (length level))
            file
            (string-to-number line))))
#+end_src

Before we dive into the implementation of this function, let’s write a test to validate (and explain) what we expect to return:

#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit--process-entry-test ()
    (setq ha-hamacs-edit-prev-head-list '())
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:42:* A Nice Headline  :ignored:")
             '("Somefile∷ A Nice Headline  " "ha-somefile.org" 42)))

    ;; For second-level headlines, we need to keep track of its parent,
    ;; and for this, we use a global variable, which we can set for the
    ;; purposes of this test:
    (setq ha-hamacs-edit-prev-head-list '("Parent"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:** Another Headline")
             '("Somefile∷ Parent﹥ Another Headline"
               "ha-somefile.org" 73)))

    (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:*** Deep Heading")
             '("Somefile∷ Parent﹥ Subparent﹥ Deep Heading"
               "ha-somefile.org" 73)))

    (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"
                                          "Subby" "Deepsubby"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:***** Deepest Heading")
             '("Somefile∷ ... Deepest Heading"
               "ha-somefile.org" 73))))
#+end_src

We next need a regular expression to pass to =s-match= to parse the output:
#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-rx-ripgrep
    (rx (group (one-or-more (not ":"))) ":"   ; filename
        (group (one-or-more digit)) ":"       ; line number
        (group (one-or-more "*"))             ; header asterisks
        (one-or-more space)
        (group (one-or-more (not ":"))))      ; headline without tags
    "Regular expression of ripgrep default output with groups.")
#+end_src

The =—new-heading= function will /prepend/ the name of the file and its parent headlines (if any) to the headline to be more useful in both understanding the relative context of the headline, as well as better to search using fuzzy matching.

This /context/ is especially important as =completing-read= will place the most recent choices at the top.

I found the use of =setf= to be quite helpful in manipulating the list of parents. Remember a =list= in a Lisp, is a /linked list/, and we can easily replace one or more parts, by pointing to an new list. This is my first iteration of this function, and I might come back and simplify it.

Essentially, if we get to a top-level headline, we set the list of parents to a list containing that new headline. If we get a second-level headine, =B=, and our parent list is =A=, we create a list =’(A B)= by setting the =cdr= of =’(A)= to the list =’(B)=. The advantage of this approach is that if the parent list is =’(A C D)=, the =setf= works the same, and the dangled /sublist/, =’(C D)= gets garbage collected.

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--new-heading (file head level)
    "Return readable entry from FILE and org headline, HEAD.
  The HEAD headline is, when LEVEL is greater than 1,
  to include parent headlines. This is done by storing
  the list of parents in `ha-hamacs-edit-prev-head-list'."
    ;; Reset the parent list to include the new HEAD:
    (cond
      ((= level 1)
       (setq ha-hamacs-edit-prev-head-list (list head)))
      ((= level 2)
       (setf (cdr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 3)
       (setf (cddr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 4)
       (setf (cdddr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 5)
       (setf (cddddr ha-hamacs-edit-prev-head-list) (list head))))
      ;; Let's never go any deeper than this...

    (format "%s∷ %s"
     (ha-hamacs-edit--file-title file)
     (s-join "﹥ " ha-hamacs-edit-prev-head-list)))
#+end_src

The following test should pass some mustard and explain how this function works:
#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit--new-heading-test ()
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Apples" 1)
             "Foobar∷ Apples"))
    (setq ha-hamacs-edit-prev-head-list '("Apples"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Oranges" 2)
             "Foobar∷ Apples﹥ Oranges"))
    (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Bananas" 3)
             "Foobar∷ Apples﹥ Oranges﹥ Bananas"))
    (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges" "Bananas"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Cantaloupe" 4)
             "Foobar∷ Apples﹥ Oranges﹥ Bananas﹥ Cantaloupe")))
#+end_src

I store the current list of parents, in the following (/gasp/) /global variable/:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-prev-head-list '("" "")
    "The current parents of headlines as a list.")
#+end_src

I would like to make the /filename/ more readable, I use the =s-match= again, to get the groups of a regular expression, remove all the dashes, and use =s-titleize= to capitalize each word:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--file-title (file)
    "Return a more readable string from FILE."
    (s-with file
      (s-match ha-hamacs-edit-file-to-title)
      (second)
      (s-replace "-" " ")
      (s-titleize)))

  (defvar ha-hamacs-edit-file-to-title
    (rx (optional (or "README-" "ha-"))
        (group (one-or-more any)) ".org")
    "Regular expression for extracting the interesting part of a
  file to use as a title.")
#+end_src

So the following tests should pass:

#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit-file-title-test ()
    (should (equal (ha-hamacs-edit-file-title "ha-apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "README-apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "README.org") "Readme")))
#+end_src

* Keybindings
With a lovely collection of functions, we need to have a way to easily call them. I’ve been using the =pretty-hydra= feature of major-mode-hydra:

#+begin_src emacs-lisp
  (pretty-hydra-define org-babel (:color blue)
    ("Navigate"
     (("g" avy-jump-org-block "goto")
      ("j" org-next-block "previous block" :color pink)
      ("k" org-previous-block "next block" :color pink))
     "Evaluate"
     (("o" avy-org-babel-execute-src-block "block")
      ("h" org-babel-execute-subtree "section")
      ("b" org-babel-execute-buffer "buffer"))
     "Tangle"
     (("t" org-babel-tangle "to File")
      ("T" org-babel-detangle "from File"))
     "Misc"
     (("e" avy-org-babel-edit-src-block "edit"))))
#+end_src

And tie this hydra into the existing leader system:
#+begin_src emacs-lisp
  (ha-leader :keymaps 'org-mode-map "o s" '("babel" . org-babel/body))
#+end_src
* Technical Artifacts                           :noexport:
Let's =provide= a name so we can =require= this file:

#+begin_src emacs-lisp :exports none
  (provide 'ha-org-literate)
  ;;; ha-org-literate.el ends here
#+end_src

#+DESCRIPTION: literate programming in Emacs Org Files.

#+PROPERTY:    header-args:sh :tangle no
#+PROPERTY:    header-args:emacs-lisp  :tangle yes
#+PROPERTY:    header-args    :results none :eval no-export :comments no mkdirp yes

#+OPTIONS:     num:nil toc:nil todo:nil tasks:nil tags:nil date:nil
#+OPTIONS:     skip:nil author:nil email:nil creator:nil timestamp:nil
#+INFOJS_OPT:  view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js