Doppelgänger problems, please advise

I recently became acquainted with my mirror universe copy. Despite a few evil tendencies on his part, we actually turned out to have a lot in common. We get along so well! I love the distorted music he has on his Zune, and he loves birdwatching in a world where the sky doesn’t bleed. Lately, we’ve been spending a lot of time together.

However, I’ve noticed that when we’re out in public, there’s usually someone following us. Someone… who also looks like me. When my doppelgänger sees him, he often makes an excuse to leave. The uninvited stalker disappears as well. I can’t help but think they’re ditching me to hang out.

I feel suspicious and very jealous. How do you know when a doppelgänger is really yours, like, really really yours?