VoiceOver alternating between very fast and slow speech in MacOS Sonoma

Hey guys, I just updated from Ventura to Sonoma, and I am not sure if what I am encountering is a bug or if there's a way to fix it somehow. I tried looking around the forum but nothing pops out as useful.
So I'm using Eloquence as my default voice, and my rate is set to 80%. However, the speech rate goes down to 50% when reading certain words. For example, Eloquence is reading back this text to me with 50%, but when I go around the website it's back to my usual 80%.
Same thing with other apps. I went looking around in VO utilities and when I VO arrow left and right some things are being read normally and some slower. It's not consistent so far, or at least I can't find the consistency.
Another thing which might be of note, I tried to switch the default voice from Eloquence to something else through the rotor, and the fast speech is spoken by the new voice (tried with Samantha and Alex until now) however Eloquence is still speaking when VO slows down to 50%. So for example, I press alt tab and it will result in Samantha reading out Voice Utility with 80%, but when I let go VO slows down and Eloquence says Voice Utility Window.
Any help for this? It's really annoying.