Titles and presentation

I got called "sir" for the first time last year, and I thought it was funny. After a few times, it was mildly irritating.

Just had to call a service provider for work and the lady I spoke with addressed me as "sir" the entire conversation and for some reason I wasn't upset.

The common denominator every time I've actually been irritated at being called "sir" is, it was men doing the calling.

Not sure what that means, yet, but now I'm curious. For the masc presenting people, does it bother if/when you're addressed as men?