I feel anxiety while taking pictures in public. I keep wondering if I’m holding the camera correctly or being judged or a zillion other things and end up just snapping a picture and moving on instead of enjoying it. I am not able to stay a bit more and try something new.
Has anyone else experienced the same and how do I overcome this

You need to accept that it is perfectly ok to feel self conscious (Nervous etc). Some also feel awkward and it can also be distracting especially when you being there as a photographer gets you noticed.

Personally I found my initial experiences awkward and had several questions running in my head like
what will they think
am I invading privacy

In short trying to avoid this scenario is what took furthered my interest in Candid Photography. I found it easier to be discrete, use a long lens and get some really good moments of unsuspecting friends and strangers.

As I started off professionally this did become a hindrance to me, and it did take me some time to get the fact that I was asked (sometimes paid) to do this job. I could interfere / step in between (and thanks to my prior experience) I knew to anticipate to take that step inside a group of people click and scoot (sometimes with a smile / acknowledgement of some sort)

The biggest insecurity one can have is "my photo is not good" . If we did mess up and people notice all it takes is the courage to acknowledge our mistake correct it (retake a pic).
Yet it also helps to stop being over critical of oneself.

Another tip - stop worrying while you are clicking. go back and in your comfort review your pictures and make a mental note of what went wrong (even professionals do make mistakes at times) and try to do a better job at the next opportunity you have

These are some suggestions based on my experiences. You can try these out and don't get disheartened if you still feel anxious, it just takes a while to figure out what works for you.