Sadly I no longer have the card so I can't re-install the card to remove the nvidia drivers properly. I generated a xorg.conf and put it in /etc/X11/xorg.conf, and it makes no mention of nvidia. My ~/.local/share/xorg/Xorg.0.log also does not mention nvidia. So it's quite strange that the system appears to be still using the nvidia drivers, any other ways I might be able to force disable the nvidia drivers?
Or worst-case scenario is there a clean way to re-install the operating system hopefully with all traces of nvidia removed. I already have a seperate /home partition.
This didn't catch all nvidia related packages, changing it out for sudo apt remove --purge *nvidia* && sudo apt autoremove got some more, though after a reboot nothing else changed.
Thanks for everyones help in trying to resolve this issue. In case anyone else is running into a similar issue in the future here is my advice:
Don't bother trying to fix this, make sure your /home is on a different partition.
Re-install with an install USB and make sure you don't nuke your home partition.
On the new system jump to a TTY and orbital strike the auto-created /home
Mount your saved partition to /home and add it to your fstab.
Re-install a couple of apps
Go on with the rest of your life, trying to preserve the install is not worth it.