Buddhism is both religion and not religion. Most of the core ideas are not metaphysical, and most modern (certainly western) practitioners don’t engage with the metaphysical (what most would consider to be “religious”) aspects at all.

Having come from a conservative Christian religious background (and finding I was wholly incompatible with it), I avoided anything even remotely resembling religion for most of my 20s.

It wasn’t until my mid 30s that I went down the Buddhism rabbit hole and what I found was entirely unlike what I expected based on my former experience with western religion.

There are also many forms of a Buddhism with varying positions e.g. Zen Buddhism is explicitly non-religious, tapping into the core insights without asking the practitioner to believe anything.