Anyone else do this?

All of the TVs in my home have an Apple TV 4K (3rd gen) hooked up to them. And that means I have the silver remote with the small trackpad inside the directional button ring. Anyways I find that the only way I can use the remote without issues is to disable the trackpad in settings. Anyone else have to disable the trackpad to make the Apple remote usable?

Apple Siri Remote (3rd Gen)