The starting assumption of a secular meditator (like me) usually is that we want to develop some abilities. You are you, just better.
If you meditate a lot, or very long time, there comes a point where you realize that it's not just developing cognitive abilities. Your motivations and perspective change as well. It's not just you doing meditation practice. The practice is also working on you. Eventually you are going towards direction where you don't want to go and it can create lots of negative feelings, including fear and hopelessness (sometimes called the Dark Night of the Soul). This is the end of the road for many people. They lose the interest and do something else.
I think it's easier for people of religion because they have innate trust that everything is going towards something better and others have done the same.