Title bar height

I'd like to resize the title bar cause it looks a bit bulky on some applications (e.g.: spotify, vs code, alacritty). I tried creating a gtk.css file as such:

headerbar entry,    
headerbar spinbutton,    
headerbar button,    
headerbar separator {    
    margin-top: 0px; 
    margin-bottom: 0px;    

headerbar {    
     min-height: 26px;    
     padding-left: 2px; 
     padding-right: 2px;    
     margin: 0px;
     padding: 0px;    

And it works great on the apps but the problem is it makes the file explorer look wonky because the top buttons are so big that I guess it needs the large title bar size. Anyone has a fix for this?

The buttons are padded based on the Icon size. You may try using an icon set with smaller icons or try disabling icons in the titlebar.
This is one of the many flaws with CSD's.
There is no easy fix for Gnomes notebook stack.

The thing is when I deleted the css file I created, the file explorer went back to how it was initially, but the rest of the apps' title bars remained thin. It only reverts back completely when I log out and back in, is there a way to make this permanent, so that the modification works on everything but not the file explorer?

Yes, it would be if you do not delete the custom .css file.

Yes but as I said deleting the file reverts the title bar height on all apps after restarting, I'm asking if I could have it changed on all apps EXCEPT the file explorer.

Yes, deleting the file that uses the snippet of code will revert everything to how it is without the file.
What am I missing, here?

You O.P. states that is what happened when using the file. It sounded like a complaint you wanted fixed.