Thank you for your reply! This really helped me! I didn't enable the killswitch as I only use the free version but it seems that there was a VPN connection still running in the background.
I identified the VPN connection using the command:
nmcli connection show --active
Looked for any connections that begin with prefix pvpn- This usually includes pvpn-killswitch and pvpn-ipv6leak-protection, and may include pvpn-routed-killswitch. Deleted all these connections using the following command:
nmcli connection delete [connection name]