Its been my observation that most moderate to heavy users have similar reactions eventually - myself included. I gave it up for years. Eventually, I found my way back and was determined to get through the anxiety/unpleasantness. What worked for me was starting back slow (2.5 mg of edibles) and working my way up from there.

Some times it was great, sometimes I could feel the anxiety/panic attack coming. In cases where the latter happened, I would really try to focus on why it was happening and what feelings I was having (mentally/physically). At a low dose, it was much easier to sort out the "who, what, when, where, why" of why the experience turned negative. I started recognizing the onset of the negative experience and learned to simply 'ride the wave' past it. And so I learned a big part of that was the expectation that it was going to be bad and everything that came with it. It became self fulfilling. As soon as I made those internal connections, it's been amazing. The only problems I have now are tolerance related :)

Any way, if you enjoyed it in the past, I'd encourage you to try it again at some point, but start with very small doses and work your way up.