It's definitely an issue in my life because no one I know is even remotely interested or understanding of collecting. If it was sports shit or something like that everyone would be cool but show them a Star Wars Hot Toy and automatically get hit in the face with the judgement stick.
I actually fear people coming over to my house because I just don't want to have that conversation. In fact my house is pretty much off limits because of it. My in-laws mock my collection on the few occasions they've been here, my friends bust balls over it, and everyone just gives me this "....really?" look. Also if my buddies come over I don't want them or their kids touching anything. Was showing a buddy some MP transformer, he was faking interest then hands it off to his 6 year old kid who had full on gross kid cheeto grime hands. I don't want people judging my shit and especially not touching my shit.
The worst is when I have someone over to do house repairs. My collection is in my garage so 90% of anything that needs to be repaired is in there. Have had many an awkward conversation like "no sir, I don't know where the hell main gas line valve is or what kind of wrench to use but will this lightsaber help?".