AppleTV Remote Issue

Hi All, Wall of text alert:

So I'm having a weird issue with my AppleTV and its remote. Currently, the only accessibility feature I have enabled on my AppleTV is Voiceover. Randomly, while navigating my homescreen or within any app, when I press an arrow button the remote will play a tri-tone (similar to the old tone used on Iphones when moving apps by hold and drag), and then continuously repeat that arrow press. Let's take the right arrow as an example. If this happens while I'm on the homescreen in row 1, column 1, the navigation will continue jumping apps until it hits row 1, column 6, then stop as it's reached the right edge of the screen. If this occurs during video playback, it will just continuously fastforward. This happens with the most recent USB-C based remote, as well as the previous lightning-based version, and on AppleTVs from HD through 4K latest inclusive. As far as I remember, this has been happening since at least TVOS 17.0, but might've been earlier.
At first I thought this might just be a dirty remote, but the fact that it's happened with at least 3 different remotes makes me think otherwise. Once the arrow repeat is triggered, the only way to regain control of my remote is to either unplug my AppleTV, or use the remote app on my phone to scroll to settings and fully restart the device. Putting it to sleep via holding the remotes power button does not fix it. The repeat just resumes once it's awakened.

Has anybody else experienced anything like this? I'm kind of hoping that this is just some kind of key repeat setting I keep activating by a jesture I'm not aware of, and therefore there's another jesture to turn it off. Because having to fully reboot every time this happens is seriously annoying.