Can't shutdown eglot

[timlod]

Config

• Server used: pylsp
• Emacs version: GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars) of 2021-03-28, modified by Debian
• Operating system: Debian 11
• Eglot version: 20211215.2131
• Eglot installation method: package.el
• Using Doom: No

Explanation

I ran into some eglot-related bug:

error in process filter: forward-sexp: Scan error: "Containing expression ends prematurely", 87, 88
error in process filter: Scan error: "Containing expression ends prematurely", 87, 88
Error running timer: (scan-error "Containing expression ends prematurely" 87 88) [2 times]

which froze my entire python buffer. I didn't mean to report this bug, but just restart eglot and continue with my work, so I attempted to do that. Unfortunately the shutdown didn't work, so this bug report is solely about the shutdown. I wouldn't know how to reproduce this state because I'm not sure how I got into the first error in the first place.

What I would like though, is some guidance/method on how to recover from such a severe error related to eglot. In my case, I don't see a way to stop eglot, and I can't get another working buffer of the file I was working on - restarting Emacs was the only thing I found that worked.

LSP transcript - M-x eglot-events-buffer (mandatory unless Emacs inoperable)

Unfortunately I killed all eglot-related buffers trying to get out of the situation (which didn't help), so I don't have this. I imagine this error is rare, so I won't try to reproduce right now lest I also lose the backtrace. Will update if it happens again.

Backtrace (mandatory, unless no error message seen or heard):

Debugger entered--Lisp error: (error "Process EGLOT (working/python-mode) not runn...")
  process-send-string(#<process EGLOT (working/python-mode)> "Content-Length: 61\15\n\15\n{\"jsonrpc\":\"2.0\",\"id\":2074,\"...")
  #f(compiled-function (arg1 &rest rest) "Send MESSAGE, a JSON object, to CONNECTION." #<bytecode 0x158293a7b2dd>)(#<eglot-lsp-server eglot-lsp-server-158293c970e8> :id 2074 :method :shutdown :params nil)
  apply(#f(compiled-function (arg1 &rest rest) "Send MESSAGE, a JSON object, to CONNECTION." #<bytecode 0x158293a7b2dd>) #<eglot-lsp-server eglot-lsp-server-158293c970e8> (:id 2074 :method :shutdown :params nil))
  jsonrpc-connection-send(#<eglot-lsp-server eglot-lsp-server-158293c970e8> :id 2074 :method :shutdown :params nil)
  jsonrpc--async-request-1(#<eglot-lsp-server eglot-lsp-server-158293c970e8> :shutdown nil :success-fn #f(compiled-function (result) #<bytecode 0x158293922719>) :error-fn #f(compiled-function (jsonrpc-lambda-elem626) #<bytecode 0x158293922745>) :timeout-fn #f(compiled-function () #<bytecode 0x158293922765>) :timeout 1.5)
  apply(jsonrpc--async-request-1 #<eglot-lsp-server eglot-lsp-server-158293c970e8> :shutdown nil :success-fn #f(compiled-function (result) #<bytecode 0x158293922719>) :error-fn #f(compiled-function (jsonrpc-lambda-elem626) #<bytecode 0x158293922745>) :timeout-fn #f(compiled-function () #<bytecode 0x158293922765>) (:timeout 1.5))
  #f(compiled-function (arg1 arg2 arg3 &rest rest) "Make a request to CONNECTION, wait for a reply.\nLike `jsonrpc-async-request' for CONNECTION, METHOD and PARAMS,\nbut synchronous.\n\nExcept in the case of a non-nil CANCEL-ON-INPUT (explained\nbelow), this function doesn't exit until anything interesting\nhappens (success reply, error reply, or timeout).  Furthermore,\nit only exits locally (returning the JSONRPC result object) if\nthe request is successful, otherwise it exits non-locally with an\nerror of type `jsonrpc-error'.\n\nDEFERRED and TIMEOUT as in `jsonrpc-async-request', which see.\n\nIf CANCEL-ON-INPUT is non-nil and the user inputs something while\nthe function is waiting, then it exits immediately, returning\nCANCEL-ON-INPUT-RETVAL.  Any future replies (normal or error) are\nignored." #<bytecode 0x158293a77601>)(#<eglot-lsp-server eglot-lsp-server-158293c970e8> :shutdown nil :timeout 1.5)
  apply(#f(compiled-function (arg1 arg2 arg3 &rest rest) "Make a request to CONNECTION, wait for a reply.\nLike `jsonrpc-async-request' for CONNECTION, METHOD and PARAMS,\nbut synchronous.\n\nExcept in the case of a non-nil CANCEL-ON-INPUT (explained\nbelow), this function doesn't exit until anything interesting\nhappens (success reply, error reply, or timeout).  Furthermore,\nit only exits locally (returning the JSONRPC result object) if\nthe request is successful, otherwise it exits non-locally with an\nerror of type `jsonrpc-error'.\n\nDEFERRED and TIMEOUT as in `jsonrpc-async-request', which see.\n\nIf CANCEL-ON-INPUT is non-nil and the user inputs something while\nthe function is waiting, then it exits immediately, returning\nCANCEL-ON-INPUT-RETVAL.  Any future replies (normal or error) are\nignored." #<bytecode 0x158293a77601>) (#<eglot-lsp-server eglot-lsp-server-158293c970e8> :shutdown nil :timeout 1.5))
  jsonrpc-request(#<eglot-lsp-server eglot-lsp-server-158293c970e8> :shutdown nil :timeout 1.5)
  eglot-shutdown(#<eglot-lsp-server eglot-lsp-server-158293c970e8> t nil nil)
  funcall-interactively(eglot-shutdown #<eglot-lsp-server eglot-lsp-server-158293c970e8> t nil nil)
  call-interactively(eglot-shutdown record nil)
  command-execute(eglot-shutdown record)
  execute-extended-command(nil "eglot-shutdown" "eglot shu")
  funcall-interactively(execute-extended-command nil "eglot-shutdown" "eglot shu")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

Minimal configuration (mandatory)

I wouldn't know how to reproduce right now, but will add this if it ever occurs again. Again, the focus here is on some method that would kill/restart eglot if something went awry (and the shutdown commands cease working), regardless of config.

[luluman]

I have encountered the same problem when using clangd and pyright.

After a few dig, I find this:

1. when running into (error "Process EGLOT (working/python-mode) not runn..."), It is hard to return to a healthy state.
2. run eglot-shutdown-all will get this error message: [jsonrpc] (warning) Sentinel for EGLOT (llvm-project/c++-mode) still hasn't run, deleting it!, and emacs will get stuck unless using c-g to break out.
3. using list-process will show some process whose prefix is "Eglot", but I can not find " EGLOT (llvm-project/c++-mode) ". Does this mean the process has already gone but does not deleted by Eglot?
4. I have checked the keys in "eglot--servers-by-project" and fond "(vc . "/workspace/llvm-project/")" is still there.

I think it may be the "jsonrpc" or "eglot" does not maintain a good context and leak some process information. I try to do some clean up on "elgot--servers-by-project" but failed. ( I do not know how to extract the process info from "eglot--servers-by-project" and check whether this server still exists, then delete or kill it)

[joaotavora]

I forgot to convert this issue worth insufficient information into a discussion back then. Doing it now.

It's fine to have a speculative conversation of course, but I can't do much debugging or spend time on this until I have a consistent repro.

[timlod]

I haven't run into it again yet - it's really hard to reproduce exactly. After the second time it happened to me, I wanted to at least document what I could gather somewhere, as it's frustrating to have to restart Emacs because eglot can't recover from this state. If I run into it again I'll report back!

[joaotavora]

Yes, that will be useful. And don't get me wrong: reports like this have positive non-zero value and are welcome.

It's just i'd like to keep the issue list clean with things that can presumably be reproduced and acted on. Your problem, for example, may lie in a personal complex emacs configuration, or and other package and we don't know.

I've been using clangd intensively myself the past two weeks with no major problems.

EDIT: fix silly autospell typos

[nemethf]

When I had a similar situation, it was probably because eglot-shutdown wasn't successful, and the file-notification handlers wasn't cleared. So, even when Eglot opened a new connection to the server, the file-notification handlers wanted to use the old, dead connection. I extracted the notification id from the backtrace and manually called file-notify-rm-watch.

Also eglot--servers-by-project contained two entries for the same major-mode/project pair.

The next time it happens, I'll try to understand how Eglot ends up in a bad state.

[joaotavora]

What do you mean file-notification handlers exactly? is their clearing not in an unwind-protect form? If not, it should be!

[nemethf]

I mean this. But the situation happened while I was working on Eglot, so I'm not saying there's bug anywhere. Anyway, next time I'll be more prepared.

[timlod]

@nemethf Can you tell me what the notification id looks like in the trace? I'd like to try at least manually righting the situation.
I keep running into this issue, but at long enough intervals that I don't see what may have caused it.
In any case eglot is unrecoverably broken for me in those cases, and I have to restart Emacs, which I'd rather not...

[nemethf]

Sorry, this was long ago, and I haven't run into this issue since then. Most probably my backtrace was a bit different. At any rate, when an file-notification event is triggered, the anonymous function is called at

eglot/eglot.el

Line 2975 in 83a61f6

((handle-event

And you can pass the value of desc to file-notify-rm-watch to remove the watch.

[t1anchen]

Hi, I encountered similar situation around this issue. I would like to post what I've found. Hopefully it could help further investigation.

Problem

When shutting down an eglot lsp server, it hang there with messaging
glitched like

Sentinel for (clangd ...) still hasn't run, deleting it!

Postmortem

Environment

Item	Description
emacs	v29.2
eglot	v1.16
clangd	v17.0.6

Reproducing

1. Emacs works on server-mode (client-server style)
2. Open a rust source file with rust-analyzer running
3. Open a c++ source file with clangd running (In my case, just
   working in a project directory under /tmp)
4. Try shutting down by eglot-shutdown-all
5. The clangd one of eglot server fell into infinite loop. Press
   C-g could jump out loop but unable to kill/terminate eglot server

Debug

By setting debug-on-exit on, here's a dump when C-g triggers

Debugger entered--Lisp error: (quit)
  accept-process-output(nil 0.1)
  #f(compiled-function (conn &optional cleanup) "Wait for JSONRPC connection CONN to shutdown.\nWith optional CLEANUP, kill any associated buffers." #<bytecode -0x1896446c2ca1c6bd>)(#<eglot-lsp-server eglot-lsp-server-1fea2e4ad6ba> t)
  apply(#f(compiled-function (conn &optional cleanup) "Wait for JSONRPC connection CONN to shutdown.\nWith optional CLEANUP, kill any associated buffers." #<bytecode -0x1896446c2ca1c6bd>) #<eglot-lsp-server eglot-lsp-server-1fea2e4ad6ba> t)
  jsonrpc-shutdown(#<eglot-lsp-server eglot-lsp-server-1fea2e4ad6ba> t)
  #f(compiled-function () #<bytecode 0xa30b1a42666bcc9>)()
  eglot-shutdown(#<eglot-lsp-server eglot-lsp-server-1fea2e4ad6ba>)
  eglot--managed-mode(-1)
  eglot--managed-mode-off()
  kill-buffer("main.cc")
  funcall-interactively(kill-buffer "main.cc")
  call-interactively(kill-buffer nil nil)
  command-execute(kill-buffer)

It is found that the jsonrpc-shutdown fell into infinite loop while
the clangd was killed/terminated on the host, though there is a
timeout setting but only working for rpc server rather than eglot
client.

Mitigation/Resolution

Wrapping the jsonrpc-shutdown with a timeout mechanism could
mitigate/resolve the issue but it might be not a permanent solution. The
root cause may require further investigation. In Emacs Lisp, there is a
macro called with-timeout to provide a mimic functionality for eglot
client side.

--- eglot.origin.el 2024-02-25 23:25:02.000000000 +0800
+++ eglot.el    2024-02-25 22:58:22.000000000 +0800
@@ -1140,7 +1140,8 @@
         (eglot--request server :shutdown nil :timeout (or timeout 1.5))
         (jsonrpc-notify server :exit nil))
     ;; Now ask jsonrpc.el to shut down the server.
-    (jsonrpc-shutdown server (not preserve-buffers))
+    (with-timeout ((or timeout 1.5) (eglot--message "JSON RPC shutting down is timeout."))
+      (jsonrpc-shutdown server (not preserve-buffers)))
     (unless preserve-buffers (kill-buffer (jsonrpc-events-buffer server)))))

 (defun eglot-shutdown-all (&optional preserve-buffers)

Reference

• https://www.gnu.org/software/emacs/manual/html_node/elisp/Timers.html#index-with_002dtimeout