panic: concurrent write to websocket connection causes dnscrypt-proxy service to restart sporadically

[paynekw]

After using the recently-added Prometheus metrics to create a dnscrypt-proxy dashboard in Grafana, I noticed that one of my instances occasionally recycles itself due to a panic causing the service to restart.

$ journalctl -u dnscrypt-proxy -e

Jul 01 17:27:20 ubuntu-16gb-nbg1-1 systemd[1]: Stopping dnscrypt-proxy.service - Encrypted/authenticated DNS proxy...
Jul 01 17:27:20 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Deactivated successfully.
Jul 01 17:27:20 ubuntu-16gb-nbg1-1 systemd[1]: Stopped dnscrypt-proxy.service - Encrypted/authenticated DNS proxy.
Jul 01 17:27:20 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Consumed 27.370s CPU time.
Jul 01 17:27:20 ubuntu-16gb-nbg1-1 systemd[1]: Started dnscrypt-proxy.service - Encrypted/authenticated DNS proxy.
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: panic: concurrent write to websocket connection
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: goroutine 26679 [running]:
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0x402246d0e0, 0x1, {0x0?, 0x402246d1a0?, 0x40024fadc8?})
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:617 +0x460
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: github.com/gorilla/websocket.(*messageWriter).Close(0x40024fae01?)
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:731 +0x48
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: github.com/gorilla/websocket.(*Conn).beginMessage(0x4026dadce0, 0x402246d1a0, 0x1)
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:480 +0x38
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: github.com/gorilla/websocket.(*Conn).NextWriter(0x4026dadce0, 0x1)
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:520 +0x40
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: github.com/gorilla/websocket.(*Conn).WriteJSON(0x5f6d00?, {0x5f6d00, 0x402246d170})
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/json.go:24 +0x2c
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: main.(*MonitoringUI).handleWebSocket.func2()
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:966 +0x330
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]: created by main.(*MonitoringUI).handleWebSocket in goroutine 26710
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 dnscrypt-proxy[2645]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:921 +0x424
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.
Jul 01 21:51:22 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Consumed 5min 26.411s CPU time, 647.3M memory peak, 0B memory swap peak.
Jul 01 21:53:22 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Scheduled restart job, restart counter is at 1.
Jul 01 21:53:22 ubuntu-16gb-nbg1-1 systemd[1]: Started dnscrypt-proxy.service - Encrypted/authenticated DNS proxy.
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: panic: concurrent write to websocket connection
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: goroutine 183713 [running]:
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0x4022403620, 0x1, {0x0?, 0x4022403710?, 0x40246bcdc8?})
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:617 +0x460
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: github.com/gorilla/websocket.(*messageWriter).Close(0x40246bce01?)
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:731 +0x48
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: github.com/gorilla/websocket.(*Conn).beginMessage(0x400b1e4840, 0x4022403710, 0x1)
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:480 +0x38
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: github.com/gorilla/websocket.(*Conn).NextWriter(0x400b1e4840, 0x1)
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:520 +0x40
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: github.com/gorilla/websocket.(*Conn).WriteJSON(0x5f6d00?, {0x5f6d00, 0x40224036e0})
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/json.go:24 +0x2c
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: main.(*MonitoringUI).handleWebSocket.func2()
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:966 +0x330
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]: created by main.(*MonitoringUI).handleWebSocket in goroutine 183697
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 dnscrypt-proxy[19043]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:921 +0x424
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.
Jul 03 13:59:15 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Consumed 44min 13.561s CPU time.
Jul 03 14:01:15 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Scheduled restart job, restart counter is at 2.
Jul 03 14:01:15 ubuntu-16gb-nbg1-1 systemd[1]: Started dnscrypt-proxy.service - Encrypted/authenticated DNS proxy.
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: panic: concurrent write to websocket connection
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: goroutine 99739 [running]:
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0x4022966450, 0x1, {0x0?, 0x4022966510?, 0x4026703dc8?})
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:617 +0x460
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: github.com/gorilla/websocket.(*messageWriter).Close(0x4026703e01?)
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:731 +0x48
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: github.com/gorilla/websocket.(*Conn).beginMessage(0x4024b838c0, 0x4022966510, 0x1)
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:480 +0x38
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: github.com/gorilla/websocket.(*Conn).NextWriter(0x4024b838c0, 0x1)
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/conn.go:520 +0x40
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: github.com/gorilla/websocket.(*Conn).WriteJSON(0x5f6d00?, {0x5f6d00, 0x40229664e0})
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/vendor/github.com/gorilla/websocket/json.go:24 +0x2c
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: main.(*MonitoringUI).handleWebSocket.func2()
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:966 +0x330
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]: created by main.(*MonitoringUI).handleWebSocket in goroutine 99737
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 dnscrypt-proxy[167369]:         /home/runner/work/dnscrypt-proxy/dnscrypt-proxy/dnscrypt-proxy/monitoring_ui.go:921 +0x424
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.
Jul 04 13:38:45 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Consumed 27min 6.187s CPU time.
Jul 04 13:40:45 ubuntu-16gb-nbg1-1 systemd[1]: dnscrypt-proxy.service: Scheduled restart job, restart counter is at 3.
Jul 04 13:40:45 ubuntu-16gb-nbg1-1 systemd[1]: Started dnscrypt-proxy.service - Encrypted/authenticated DNS proxy.

$ ./dnscrypt-proxy --version
2.1.12

$ uname -a
Linux ubuntu-16gb-nbg1-1 6.8.0-63-generic #66-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 13 20:09:49 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"

The VM is hosted at Hetzner. dnscrypt-proxy was installed right after the VM had been created and had caught up with updates, and the installation was done "by the book" i.e. by following exactly the instructions for installation on Linux provided in the official documentation. Aside from the restarts, it works as expected.

My other instance is running on a Raspberry Pi 5 Model B and doesn't seem to have the same issue. It runs non-stop for weeks, usually until I have to update or reboot the RPI as part of routine maintenance.

[paynekw]

Seems to only happen if there is a Monitoring Dashboard open in a browser tab somewhere. Make sure no browser tabs open anywhere on the Monitoring Dashboard, no problem. Gorilla like banana but only like to eat one at a time.

[62f1346c-4576-42b3-afcb-23ef1820da34]

This didn't fix it for you?

[paynekw]

Heartfelt thanks to Frank and everyone who has contributed to creating and maintaining this beautiful and useful piece of software for free, and for the benefit of the world.