I found a solution to my problem (sorta).
I reenabled "Fractional scaling" and set it to 150% this places 95% of the desktop icons in the right place.
Some of them still move due to not enough space in that certain row. I can live with that since they will still be on the same side of the desktop (so I can easly click on them by muscle memory). And 75% of the time in using my external screens.