Debugging Emacs

Emacs is made to be modified. One of my skeptical colleagues said the fact that Emacs has state scares him, but I think it's one of its strengths (and why it has endured).

The Problem

I started doing some Ruby and Rails review over the holidays and had just finished configuring Emacs for Ruby usage. While spinning up my Ruby project, Emacs became unresponsive and wouldn't open the .rb file I needed to get into. It would hang without any additional CPU usage (confirmed by htop), but none of the keybindings worked. I initially thought that something with the new packages for working with Ruby, but it became evident this was much more insidious. Upon rebooting Emacs, I couldn't open any files without the application hanging..

A Red Herring

I use Doom Emacs because it's really easy to set up and configure, and for me it had significant performance improvements over Spacemacs (I build Emacs with native compilation, which Doom supports). The suggested troubleshooting for the framework starts with running $ doom doctor, which yielded a few expected warning and no errors.

However, I did get the following line printed after exiting the process (Ctrl-C)

/var/folders/hc/${weird_hash}/T//doom.3016.0.sh: line 1: #!/usr/bin/env: No such file or directory

I started troubleshooting on the Recurse Center Zulip (which has a very active Emacs user group), and someone else confirmed that they got the same line printed after exiting doom doctor with their working installation of Emacs. This seemed to me like it wasn't actually a problem so I cast my net wider...

Nuke It From Orbit (It's the only way to be sure)

I decided to save my ~/.config/doom files and start over. I wiped my Emacs installation, and reinstalled and upgraded Emacs to 29.1 using

$ brew install emacs-plus@29 --with-native-compilation`

I then reinstalled Doom Emacs with

$ git clone --depth 1 --single-branch https://github.com/doomemacs/doomemacs ~/.config/emacs
~/.config/emacs/bin/doom install

I opened Emacs with the fresh install and tried to open the init.el file and it hung. again.

##Binary Troubleshooting Doom emacs init.el file comes with a few modules enabled by default. To continue to troubleshoot this, I started toggling and untoggling modules until I finally discovered that the vc version control module was the culprit for the hanging process upon opening any file.

Doom Emacs Help

While I'd used the debugger within Emacs, it had always been in a situation where hitting Ctrl-G enough would exit the process (and I could just use toggle-debug-on-error or debug-on-entry). Other recursers suggested I take a look at the Doom Emacs discord, which led me to the following discourse about backtracing, including a section on backtracing from frozen Emacs!

I was able to set debug-on-entry for find-file if the vc module was installed, found the active Emacs process number, and ran $ kill -USR2 $EMACSPID. Lo, I finally generated a backtrace!

The Real Problem

So I knew that with the vc module enabled and running find-file, Emacs would hang. I also discovered that running any magit command would also hang with the vc module disabled. I was able to open files, just not run magit. Some recursers suggested that I compare the backtraces for the two processes and discovered both had call-process("/opt/homebrew/bin/gpg" nil..., which suggested the issue was actually with gpg.

It turned out I had an expired gpg key, and command line troubleshooting for gpg just kept generating lockfiles. I removed the keys, and finally Emacs worked as expected!

While killing an Emacs process without killing Emacs is now in my org file, I'll probably forget all the other steps it took to get here, so I thought it was important to write down for posterity!