yeah, that doesn't sound good at all, but I heard it's a common issue on Rhinos and other Yamaha UTVs with that engine braking system.
a typical CVT shenanigans when the engine braking isn't quite right. That clicking and delay points right to wear in the primary clutch system
The one way bearing is almost certainly the main culprit. It's inside the wet clutch. When it starts to fail, it won't grab properly during engine braking, causing that clicking/slipping before it catches. If it's really bad, you'll lose engine braking altogether eventually.
You'll need to remove the CVT outer cover, then likely the inner cover for the wet clutch, to get a good look at that one way bearing. It's a bit of a process, but there are tons of videos out there for Rhinos. Get that service manual out to make it right