I ended up using this tutorial for purging everything Nvidia and reinstalled the nvidia-driver-535. Got a black screen while installing but after reboot it seems to work without any issues. I can now open the Nvidia settings and I see a lot more things in the settings plus it says "Nvidia Corporation" under Settings > About.
Thank you for all the help!