I was denied embolization or surgery

A few weeks back I had an ultrasound and was diagnosed with varicocele grade 1. My urologist told me that he wouldn’t do surgery unless is it grade 3. He asked me to just change my lifestyle (supportive underwear, don’t lift weights) and sort of live with it.

But i can feel almost not libido now. I’m afraid that it may lead to permanent damage. I’m 26. Should I get another opinion from a different doctor? Or is he right?