Update broke VNC — DKMS regression + Wayland re-enabled (kernel 7.0.0-38-generic)
System:
- Zorin OS 18.1 Core
- Intel Core i7-4790K
- NVIDIA GeForce GTX 1050 Ti (Pascal)
- Driver: nvidia-driver-580 (580.178.04)
- VNC: x11vnc, started on-demand via SSH
Symptom
After today's system update (kernel 7.0.0-38-generic), x11vnc failed to start with:
XOpenDisplay failed (:1)
VNC connection attempt from remote host failed immediately. No Xauthority file present at /run/user/1000/gdm/Xauthority. Directory did not exist.
Diagnosis
Step 1 — DKMS check
ls /lib/modules/7.0.0-38-generic/updates/dkms/ | grep nvidia
Output showed .ko files instead of .ko.zst. DKMS did not rebuild the nvidia modules correctly for the new kernel. This is a recurring pattern on this machine — has occurred on kernels 30, 31, and now 38. (That I noticed)
Step 2 — Driver state
nvidia-smi
nvidia-smi responded correctly after manual DKMS rebuild, confirming the driver itself is functional once rebuilt. However VNC still failed after reboot.
Step 3 — X session state
ps aux | grep Xorg
No Xorg process present.
nvidia-smi # check Processes table
Showed /usr/bin/Xwayland consuming GPU — not Xorg.
Root cause confirmed: The update re-enabled Wayland in GDM, overriding a previously set WaylandEnable=false in /etc/gdm3/custom.conf. x11vnc requires an active Xorg session. With Wayland active, no Xorg process runs, no Xauthority file is generated, and x11vnc cannot attach to a display.
Root Causes — Two Separate Issues
1. DKMS regression on kernel update (recurring)
Every kernel update on this machine has failed to produce .ko.zst modules for the nvidia driver. Modules build as plain .ko and must be manually rebuilt:
sudo dkms remove nvidia/$(cat /sys/module/nvidia/version) -k 7.0.0-38-generic
sudo dkms install nvidia/$(cat /sys/module/nvidia/version) -k 7.0.0-38-generic
Verify rebuild:
ls /lib/modules/7.0.0-38-generic/updates/dkms/ | grep nvidia
All five modules should show .ko.zst.
This has occurred on kernels 30, 31, and 38. The pattern suggests something in the Zorin update pipeline is consistently failing to trigger a clean DKMS rebuild for this driver/kernel combination. Worth investigating whether the package hooks are firing correctly on kernel install.
2. Wayland re-enabled silently on update
/etc/gdm3/custom.conf had WaylandEnable=false set explicitly. The system update reset this — either by overwriting the file or by reverting the setting. After reboot, GDM started an Xwayland session instead of Xorg.
Fix:
sudo nano /etc/gdm3/custom.conf
Ensure the following line is present and uncommented:
WaylandEnable=false
Reboot. GDM starts Xorg session. Xauthority file reappears at /run/user/1000/gdm/Xauthority. x11vnc starts and connects normally.
Notes
This machine runs x11vnc on-demand via SSH rather than as a systemd service — attempts to run it as a service were abandoned due to the complexity of reliably targeting a user X session at boot. The on-demand SSH approach is reliable when the underlying X session is Xorg.
The Wayland reset is the more significant issue of the two. If WaylandEnable=false is being overwritten by package updates, this will recur on every update that touches GDM. Would be useful to know whether this is a known behavior — whether custom.conf is treated as a managed config file that gets reset, or whether something else is responsible for the reversion.
Pascal-generation GPUs (GTX 10xx series) have known issues with GSP firmware loading under newer driver versions. This machine also carries /etc/modprobe.d/nvidia-power-management.conf with:
options nvidia NVreg_PreserveVideoMemoryAllocations=1 NVreg_EnableGpuFirmware=0
This disables GSP firmware loading and has resolved suspend/wake issues previously. Noted here for completeness in case it is relevant to the DKMS regression pattern.
Zorin OS 18.1 / kernel 7.0.0-38-generic / nvidia-driver-580 / GTX 1050 Ti / October 6, 2026