Last year I really missed being young again and made the decision to try smoking again with a friend. I had the worst experience ever. I was in a constant state of panic, couldn't breathe, felt like I didn't exist, heart was pounding, etc. It was aweful. I will never ever do that again, lol. It's better left in the past.
Anyway, aside from my pointless story...
I think if you truely want to stop smoking pot you can do it. There is a huge difference between wanting to smoke, and needing to smoke. You need to figure out whether or not you want to or need to. If you want to, it should be easier to break the habit. If you feel you need to smoke, then Dodger is right, you should try seeking some sort of help because that is a sign of addiction. I know a lot of people think pot isn't addictive, but the lifestyle and habit definitely is.
You already know that smoking is a huge cause of anxiety, so perhaps if you quit, you wouldn't feel as anxious as you do. And just because your friends smoke, doesn't mean you have to. If they are your friends, they will understand and accept your need for change and not pressure you into doing it when they do. You just need to have strength and willpower and you can do it!
Good luck on your first day of not smoking.