The only time I hear this in person is when I'm the one saying it, to make sure that people know in a discussion about race and gender issues that while I am acutely aware of my own privilege , unconscious biases that form after decades of acclimation to that privilege are not easy to recognize and stamp out.
If someone uses it to discredit one of my points, I would think for a bit to question myself and see if there is a conscious/unconscious bias that's driving my viewpoint and if I don't see one, I'd politely ask if they can explain further.
As a white male in his 30s who has benefited from the privileges greatly, you simply must be at least cognizant that privilege has had an impact on your world view, even if you don't know how it manifests all the time in your thoughts.
Note I tend to avoid internet/online conversations that would veer into this kind of talk so I don't have to defend myself much here.