It looks like the freezes are gone after switching to alsa as output. I had lots of freezes recently and it was really annoying to listen to my fav' music. So much that I even tried to come up with alternatives such as mpd, emms and the like. I'm glad those freezes are a problem of the past now. I've been using cmus for more than a decade now and I can't imagine any better way of listening to music without any distractions.

~/.config/cmus/rc

set output_plugin=alsa
set dsp.alsa.device=default
set mixer.alsa.device=default
set mixer.alsa.channel=Master