After some research it appears that the Realtek Wifi driver was to be blamed: every time the wifi driver scans for accesspoints, the +/ 50ms latency occured. Simply disabling wifi solves the issue. In my case I use a wired connection, so I just removed the RTL8723BE mini PCIe card. Problem solved.
More background info on this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1262957 and https://bugzilla.kernel.org/show_bug.cgi?id=108461