1. Turn on VoiceOver.
2. Open Safari on any website of more than a couple pages and scroll down to the bottom so that you're viewing a part of the web page that does not contain VoiceOver focus.
3. Use the shortcut to toggle Invert Colors (such as VO+Tab, VO+Command+8).
After the colors invert, Safari scrolls to the top of the web page. More accurately, it scrolls to the part of the page containing VoiceOver focus.
This seems like a bug because changing the setting of Invert Colors should not cause Safari to take any such action. It's probably minor, as only low-vision users who also use VoiceOver would be likely to encounter it.
I'm on Big Sur 11.1 with Safari v14.0.2.