You should not need to remove Gnome. That said, I have installed XFCE on Zorin 16 Core and I do not see gnome processes running unless I start them. If I kill the process I start, it stays stopped. So I am not sure on this.
There are some services that are Base to the System. This would be as true on Xubuntu, as on Zorin OS Core with XFCE installed.
Installing some D.E.'s can add things you don't expect. Mate or KDE can add quite a lot - installing Mate usually brings a start up sound you did not have before.
But chaos? No... I would not say that at all.
Admittedly, when I remove either Mate or KDE, I remove all of it by using Synaptic, instead of using the terminal.