#+title:  Configuring Python in Emacs
#+author: Howard X. Abrams
#+date:   2021-11-16
#+tags: emacs python programming

import re

A literate programming file for configuring Python.

#+begin_src emacs-lisp :exports none
  ;;; ha-programming-python --- Python configuration. -*- lexical-binding: t; -*-
  ;;
  ;; © 2021-2023 Howard X. Abrams
  ;;   Licensed under a Creative Commons Attribution 4.0 International License.
  ;;   See http://creativecommons.org/licenses/by/4.0/
  ;;
  ;; Author: Howard X. Abrams <http://gitlab.com/howardabrams>
  ;; Maintainer: Howard X. Abrams
  ;; Created: November 16, 2021
  ;;
  ;; This file is not part of GNU Emacs.
  ;;
  ;; *NB:* Do not edit this file. Instead, edit the original literate file at:
  ;;            ~/src/hamacs/ha-programming-python.org
  ;;       And tangle the file to recreate this one.
  ;;
  ;;; Code:
  #+end_src
* Introduction
The critical part of Python integration with Emacs is running LSP in Python using [[file:ha-programming.org::*direnv][direnv]]. And the question to ask is if the Python we run it in Docker or in a virtual environment.

While Emacs supplies a Python editing environment, we’ll still use =use-package= to grab the latest:
#+begin_src emacs-lisp
  (use-package python
    :after flycheck
    :mode ("[./]flake8\\'" . conf-mode)
    :mode ("/Pipfile\\'" . conf-mode)
    :init
    (setq python-indent-guess-indent-offset-verbose nil
          flycheck-flake8-maximum-line-length 120)
    :config
    (setq python-shell-interpreter (or (executable-find "ipython") "python"))

    (flycheck-add-next-checker 'python-pylint 'python-pycompile 'append))
#+end_src

** Keybindings
Instead of memorizing all the Emacs-specific keybindings, we use [[https://github.com/jerrypnz/major-mode-hydra.el][major-mode-hydra]] defined for =python-mode=:

#+BEGIN_SRC emacs-lisp
  (use-package major-mode-hydra
    :after python
    :config
    (defvar ha-python-eval-title (font-icons 'mdicon "run" :title "Python Evaluation"))
    (defvar ha-python-goto-title (font-icons 'faicon "python" :title "Python Symbol References"))
    (defvar ha-python-refactor-title (font-icons 'faicon "recycle" :title "Python Refactoring"))

    (pretty-hydra-define python-evaluate (:color blue :quit-key "C-g"
                                          :title ha-python-eval-title)
      ("Section"
       (("f" python-shell-send-defun "Function/class")
        ("e" python-shell-send-statement "Line")
        (";" python-shell-send-string "Expression"))
       "Entirety"
       (("f" python-shell-send-file "File")
        ("b" python-shell-send-buffer "Buffer")
        ("r" elpy-shell-send-region-or-buffer "Region"))))

    (pretty-hydra-define python-refactor (:color blue :quit-key "C-g"
                                          :title ha-python-refactor-title)
      ("Simple"
       (("r" iedit-mode "Rename"))
       "Imports"
       (("A" python-add-import "Add Import")
        ("a" python-import-symbol-at-point "Import Symbol")
        ("F" python-fix-imports "Fix Imports")
        ("S" python-sort-imports "Sort Imports"))))

    (pretty-hydra-define python-goto (:color pink :quit-key "C-g"
                                      :title ha-python-goto-title)
      ("Statements"
       (("s" xref-find-apropos "Find Symbol" :color blue)
        ("j" python-nav-forward-statement "Next")
        ("k" python-nav-backward-statement "Previous"))
       "Functions"
       (("F" imenu "Jump Function" :color blue)
        ("f" python-nav-forward-defun "Forward")
        ("d" python-nav-backward-defun "Backward")
        ("e" python-nav-end-of-defun "End of" :color blue))
       "Blocks"
       (("u" python-nav-up-list "Up" :color blue)
        (">" python-nav-forward-block "Forward")
        ("<" python-nav-backward-block "Backward"))))

    (major-mode-hydra-define python-mode (:quit-key "C-g" :color blue)
      ("Server"
       (("S" run-python "Start Server")
        ("s" python-shell-switch-to-shell "Go to Server"))
       "Edit"
       (("r" python-refactor/body "Refactor...")
        (">" python-indent-shift-left "Shift Left")
        ("<" python-indent-shift-right "Shift Right"))
       "Navigate/Eval"
       (("e" python-evaluate/body "Evaluate...")
        ("g" python-goto/body "Go to..."))
       "Docs"
       (("d" python-eldoc-at-point "Docs on Symbol")
        ("D" python-describe-at-point "Describe Symbol")))))
#+end_src

Sections below can add to this with =major-mode-hydra-define+=.

Note: Install the following packages /globally/ for Emacs:
#+begin_src sh
  pip install flake8 flake8-bugbear pylint pyright mypy pycompile black ruff ipython
#+end_src

But certainly add those to each project’s =requirements-dev.txt= file.

iPython has a feature of [[https://ipython.readthedocs.io/en/stable/config/intro.html#python-configuration-files][loading code on startup]] /per profile/. First, create it with:

#+BEGIN_SRC sh
  ipython profile create
#+END_SRC

Next, after reading David Vujic’s [[https://davidvujic.blogspot.com/2025/03/are-we-there-yet.html][Are We There Yet]] essay, I took a look at [[https://github.com/DavidVujic/my-emacs-config?tab=readme-ov-file#python-shell][his Python configuration]], and added the /auto reloading/ feature to the iPython /profile configuration/:

#+BEGIN_SRC python :tangle ~/.ipython/profile_default/ipython_config.py
  c = get_config()  #noqa

  %load_ext autoreload
  %autoreload 2

  # c.InteractiveShellApp.extensions = ['autoreload']
  # c.InteractiveShellApp.exec_lines = ['%autoreload 2']
#+END_SRC

** Virtual Environment
When you need a particular version of Python, use [[https://github.com/pyenv/pyenv][pyenv]] globally:
#+begin_src sh
  pip install pyenv
#+end_src

And have this in your =.envrc= file for use with [[file:ha-programming.org::*Virtual Environments with direnv][direnv]]:
#+begin_src conf
  use python 3.7.1
#+end_src

Also, you need the following in your =~/.config/direnv/direnvrc= file (which I have):
#+begin_src shell
  use_python() {
    local python_root=$(pyenv root)/versions/$1
    load_prefix "$python_root"
    if [[ -x "$python_root/bin/python" ]]; then
      layout python "$python_root/bin/python"
    else
      echo "Error: $python_root/bin/python can't be executed."
      exit
    fi
  }
#+end_src
** Editing Python Code
Let’s integrate this [[https://github.com/wbolster/evil-text-object-python][Python support for evil-text-object]] project:
#+begin_src emacs-lisp
  (when (fboundp 'evil-define-text-object)
    (use-package evil-text-object-python
      :hook (python-mode . evil-text-object-python-add-bindings)))
#+end_src
This allows me to delete a Python “block” using ~dal~.
** Docker Environment
Docker really allows you to isolate your project's environment. The downside is that you are using Docker and probably a bloated container. On my work laptop, a Mac, this creates a behemoth virtual machine that immediately spins the fans like a wind tunnel.

But, but... think of the dependencies!

Enough of the rant (I go back and forth), after getting Docker installed and running (ooo Podman ... shiny), and you've created a =Dockerfile= for your project, let's install [[https://github.com/snbuback/container-env][container-env]].

Your project's =.envrc= file would contain something like:
#+begin_src shell
  CONTAINER_NAME=my-docker-container
  CONTAINER_WRAPPERS=(python3 pip3 yamllint)
  CONTAINER_EXTRA_ARGS="--env SOME_ENV_VAR=${SOME_ENV_VAR}"

  container_layout
#+end_src
** Unit Tests
#+begin_src emacs-lisp
  (use-package python-pytest
    :after python
    :commands python-pytest-dispatch
    :init
    (use-package major-mode-hydra
      :config
      (defvar ha-python-tests-title (font-icons 'devicon "pytest" :title "Python Test Framework"))
      (pretty-hydra-define python-tests (:color blue :quit-key "q"
                                                :title ha-python-tests-title)
        ("Suite"
         (("a" python-pytest "All")
          ("f" python-pytest-file-dwim "File DWIM")
          ("F" python-pytest-file "File"))
         "Specific"
         (("d" python-pytest-function-dwim "Function DWIM")
          ("D" python-pytest-function "Function"))
         "Again"
         (("r" python-pytest-repeat "Repeat tests")
          ("p" python-pytest-dispatch "Dispatch"))))

      (major-mode-hydra-define+ python-mode (:quit-key "q" :color blue)
        ("Misc"
         (("t" python-tests/body "Tests..."))))))
#+end_src
* Elpy
The [[https://elpy.readthedocs.io/en/latest/introduction.html][Elpy Project]] expands on the =python-mode=.

#+BEGIN_SRC emacs-lisp
  (use-package elpy
    :ensure t
    :init
    (elpy-enable))
#+END_SRC

Let’s expand our =major-mode-hydra= with some extras:

#+begin_src emacs-lisp
  (use-package major-mode-hydra
    :after elpy
    :config

    (pretty-hydra-define python-evaluate (:color blue :quit-key "q"
                                          :title ha-python-eval-title)
      ("Section"
       (("F" elpy-shell-send-defun "Function")
        ("E" elpy-shell-send-statement "Statement")
        (";" python-shell-send-string "Expression"))
       "Entirety"
       (("B" elpy-shell-send-buffer "Buffer")
        ("r" elpy-shell-send-region-or-buffer "region"))
       "And Step..."
       (("f" elpy-shell-send-defun-and-step "Function" :color pink)
        ("e" elpy-shell-send-statement-and-step "Statement" :color pink))))

    (pretty-hydra-define+ python-refactor nil
      ("Elpy"
       (("r" elpy-refactor-rename "Rename")
        ("i" elpy-refactor-inline "Inline var")
        ("v" elpy-refactor-extract-variable "To variable")
        ("f" elpy-refactor-extract-function "To function")
        ("a" elpy-refactor-mode "All..."))))

    (major-mode-hydra-define+ python-mode (:quit-key "q" :color blue)
      ("Server"
       (("s" elpy-shell-switch-to-shell "Go to Server")
        ("C" elpy-config "Config Elpy"))
       "Edit"
       (("f" elpy-black-fix-code "Fix/format code"))
       "Docs"
       (("d" elpy-eldoc-documentation "Describe Symbol")
        ("D" elpy-doc "Docs Symbol")))))
#+end_src

* LSP Integration of Python
** Dependencies
Each Python project's =requirements-dev.txt= file would reference the [[https://pypi.org/project/python-lsp-server/][python-lsp-server]] (not the /unmaintained/ project, =python-language-server=):

#+begin_src conf :tangle no
  python-lsp-server[all]
#+end_src

*Note:* This does mean, you would have a =tox.ini= with this line:
#+begin_src conf
  [tox]
  minversion = 1.6
  skipsdist = True
  envlist = linters
  ignore_basepython_conflict = True

  [testenv]
  basepython = python3
  install_command = pip install {opts} {packages}
  deps = -r{toxinidir}/test-requirements.txt
  commands = stestr run {posargs}
             stestr slowest
  # ...
#+end_src
** Pyright
I’m using the Microsoft-supported [[https://github.com/Microsoft/pyright][pyright]] package instead. Adding this to my =requirements.txt= files:
#+begin_src conf :tangle no
  pyright
#+end_src

The [[https://github.com/emacs-lsp/lsp-pyright][pyright package]] works with LSP.

#+begin_src emacs-lisp :tangle no
  (use-package lsp-pyright
      :hook (python-mode . (lambda () (require 'lsp-pyright)))
      :init (when (executable-find "python3")
                (setq lsp-pyright-python-executable-cmd "python3")))
#+end_src
*** Keybindings
Now that the [[file:ha-programming.org::*Language Server Protocol (LSP) Integration][LSP Integration]] is complete, we can stitch the two projects together, by calling =lsp=. I oscillate between automatically turning on LSP mode with every Python file, but I sometimes run into issues when starting, so I conditionally turn it on.

#+begin_src emacs-lisp
  (defvar ha-python-lsp-title (font-icons 'faicon "python" :title "Python LSP"))

  (defun ha-setup-python-lsp ()
    "Configure the keybindings for LSP in Python."
    (interactive)

    (pretty-hydra-define python-lsp (:color blue :quit-key "q"
                                     :title ha-python-lsp-title)
      ("Server"
       (("D" lsp-disconnect "Disconnect")
        ("R" lsp-workspace-restart "Restart")
        ("S" lsp-workspace-shutdown "Shutdown")
        ("?" lsp-describe-session "Describe"))
       "Refactoring"
       (("a" lsp-execute-code-action "Code Actions")
        ("o" lsp-organize-imports "Organize Imports")
        ("l" lsp-avy-lens "Avy Lens"))
       "Toggles"
       (("b" lsp-headerline-breadcrumb-mode "Breadcrumbs")
        ("d" lsp-ui-doc-mode "Documentation Popups")
        ("m" lsp-modeline-diagnostics-mode "Modeline Diagnostics")
        ("s" lsp-ui-sideline-mode "Sideline Mode"))
       ""
       (("t" lsp-toggle-on-type-formatting "Type Formatting")
        ("h" lsp-toggle-symbol-highlight "Symbol Highlighting")
        ("L" lsp-toggle-trace-io "Log I/O"))))

    (pretty-hydra-define+ python-goto (:quit-key "q")
      ("LSP"
       (("g" lsp-find-definition "Definition")
        ("d" lsp-find-declaration "Declaration")
        ("r" lsp-find-references "References")
        ("t" lsp-find-type-definition "Type Definition"))
       "Peek"
       (("D" lsp-ui-peek-find-definitions "Definitions")
        ("I" lsp-ui-peek-find-implementation "Implementations")
        ("R" lsp-ui-peek-find-references "References")
        ("S" lsp-ui-peek-find-workspace-symbol "Symbols"))
       "LSP+"
       (("u" lsp-ui-imenu "UI Menu")
        ("i" lsp-find-implementation "Implementations")
        ("h" lsp-treemacs-call-hierarchy "Hierarchy")
        ("E" lsp-treemacs-errors-list "Error List"))))

    (major-mode-hydra-define+ python-mode nil
      ("Server"
       (("l" python-lsp/body "LSP..."))
       "Edit"
       (("r" lsp-rename "Rename")
        ("=" lsp-format-region "Format"))
       "Navigate"
       (("A" lsp-workspace-folders-add "Add Folder")
        ("R" lsp-workspace-folders-remove "Remove Folder"))
       "Docs"
       (("D" lsp-describe-thing-at-point "Describe LSP Symbol")
        ("h" lsp-ui-doc-glance "Glance Help")
        ("H" lsp-document-highlight "Highlight"))))

    (call-interactively 'lsp))

  (use-package lsp-mode
    :config
    (major-mode-hydra-define+ python-mode (:quit-key "q")
      ("Server"
       (("L" ha-setup-python-lsp "Start LSP Server")))))

  ;; ----------------------------------------------------------------------
  ;; Missing Symbols to be integrated?
  ;; "0" '("treemacs" . lsp-treemacs-symbols)
  ;; "/" '("complete" . completion-at-point)
  ;; "k" '("check code" . python-check)
  ;; "Fb" '("un-blacklist folder" . lsp-workspace-blacklist-remove)
  ;; "hs" '("signature help" . lsp-signature-activate)
  ;; "tT" '("toggle treemacs integration" . lsp-treemacs-sync-mode)
  ;; "ta" '("toggle modeline code actions" . lsp-modeline-code-actions-mode)
  ;; "th" '("toggle highlighting" . lsp-toggle-symbol-highlight)
  ;; "tl" '("toggle lenses" . lsp-lens-mode)
  ;; "ts" '("toggle signature" . lsp-toggle-signature-auto-activate)
#+end_src
* Project Configuration
I work with a lot of projects with my team where I need to /configure/ the project such that LSP and my Emacs setup works. Let's suppose I could point a function at a project directory, and have it /set it up/:

#+begin_src emacs-lisp
  (defun ha-python-configure-project (proj-directory)
    "Configure PROJ-DIRECTORY for LSP and Python."
    (interactive "DPython Project: ")

    (let ((default-directory proj-directory))
      (unless (f-exists? ".envrc")
        (message "Configuring direnv")
        (with-temp-file ".envrc"
          ;; (insert "use_python 3.7.4\n")
          (insert "layout_python3\n"))
        (direnv-allow))

      (unless (f-exists? ".pip.conf")
        (message "Configuring pip")
        (with-temp-file ".pip.conf"
          (insert "[global]\n")
          (insert "index-url = https://pypi.python.org/simple\n"))
        (shell-command "pipconf --local")
        (shell-command "pip install --upgrade pip"))

      (message "Configuring pip for LSP")
      (with-temp-file "requirements-dev.txt"
        (insert "python-lsp-server[all]\n")

        ;; Let's install these extra packages individually ...
        (insert "pyls-flake8\n")
        ;; (insert "pylsp-mypy")
        ;; (insert "pyls-isort")
        ;; (insert "python-lsp-black")
        ;; (insert "pyls-memestra")
        (insert "pylsp-rope\n"))
      (shell-command "pip install -r requirements-dev.txt")))
#+end_src
* Major Mode Hydra
* Technical Artifacts                                :noexport:
Let's =provide= a name so we can =require= this file:

#+begin_src emacs-lisp :exports none
  (provide 'ha-programming-python)
  ;;; ha-programming-python.el ends here
  #+end_src

#+description: A literate programming file for configuring Python.

#+property:    header-args:sh :tangle no
#+property:    header-args:emacs-lisp  :tangle yes
#+property:    header-args    :results none :eval no-export :comments no mkdirp yes

#+options:     num:nil toc:t todo:nil tasks:nil tags:nil date:nil
#+options:     skip:nil author:nil email:nil creator:nil timestamp:nil
#+infojs_opt:  view:nil toc:t ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js