Hey, @Dixit, why not to try one of those proverbial "Linux black-magic" techs? 
That black-magic tech is called "zram". It can inflate RAM at the expense of CPU (tested personally on Raspberry Pi
)
What it does: zram creates a virtual swapping file in you real RAM and stores swapped data there (before they get swapped to your second swapping destination - your HDD).
But: the data are compressed in realtime by zram (thus the need of some CPU power), and that means it consumes less space in your virtual RAM-based swap file. And it is FAST (when compared to HDD) 
So it is effectively a compressed RAM buffer, acting as a swap device, consuming less RAM than would be really required. Only when this buffer is maxed out, the system starts using the "real" swap on your HDD. It consumes some CPU power to [de]compress stored data; the amount of consumed CPU cycles is based on compression algorithm selected; some algorithms are very light (lz4), others quite demanding (zstd), but those demanding ones get better compress rations, and thus allow more data to be stored to your RAM. No worries, if RPi CPU can handle it, that yours x86-based can definitely too...
I believe this might be a perfect solution for your usecase 
More info here:
...and how to setup it manually here:
Notes:
- There is more that one way how to install and enable zram on your system. The above solution is a manual setup, but I like it because I have a good control over what it does and how it behaves in the system.
- You might want to set the target higher than the above example, to e.g. 1G at least.
- I would think twice before turning off the swap. But if it is left turned on, in order for your system to use zram swap device first, before your real HHD-based swap, zram swap device must have a higher priority set. Just let me know if you decide to try this solution, I will help you further to tweak it properly
- If you test it and do not like it, the whole zram thing can be very easily disabled. So no worries to try this out, there is nothing to lose there, is it?