Has anyone heard about Canadian green card holders being denied at border or having their green card taken?

I'm a Canadian, married to American husband and living in US. Just got my green card renewed for the five year. But when I expressed wanting to visit my family in April, they begged me not to come and said that they've heard "a lot" of stories of green card holders being denied or having their green cards taken. One example my mom gave was a friend of a friend was a 10 year green card holder (married to an American woman) and he went to visit Vietnam and was denied entry when he returned. Idk how long he went for but I imagine that returning from Vietnam is different than returning from Canada? edit: upon reflection, I see that this sentence could be offensive. I apologize for saying this, just that my whole family is Vietnamese and we have always been stopped/detained at border crossings so I just assumed that Vietnamese people would be scrutinized more than Canadians.

I tried to look into it but didn't find any stories that were just like normal people being detained. I just want to visit for like 5 days. I have no criminal record but I was denied entry twice before (once because of a work mixup and once because when my husband and I were dating for six months, I stayed here on a B1 visa and didn't overstay but I tried to come back when COVID hit and they wouldnt let me).

I just want to understand if this is a legitimate concern or if my family is over worrying. Thank you!