Depends how you do it. Many years ago, walking along Byres Road in Glasgow and I see a smelly, hipster type leaning against the wall with an SLR at his hip, clearly aimed towards the foot traffic. I do photography myself, and as I passed the point where I'd be the reference, I heard the click of the shutter as I walked past.
Stopped, backwards walked up beside the guy and whispered in his ear:
'Why did you take a picture of me?'
Shat his pants, explained it was for a project, and I told him not to be so creepy about it. The more obvious you are, the less people are bothered.
Put on a hi-vis, that always makes people invisible!