If you don't use your Nvidia GPU for anything that requires 3D acceleration, I say, don't download and install Nvidia stuff. But if the tasks that you do requires 3D acceleration, like gaming for example, then your going to need those drivers.
At the end of the day, if your really worried, the logical thing to do, would be to backup your computer before you run the updates. So that way if something goes wrong, you can restore.