For every other team I have seen it is a waste of time, you either under estimate how many points something is and you get moaned at you are late or you over estimate and you are praised for delivering early.
I pretty much caveat every single estimate I give (either time or the made up points) and say "if you want accurate estimates, a good rule of thumb is that I spend 1/4 of the time estimating to planned work - so if I think it will take 4 weeks, I should spend 1 week estimating" - no one ever wants you to do the accurate estimate so when I am moaned at because the estimates are incorrect I refer back to this statement.
What is the actual goal? Is it to track a teams progress (track by actual delivered work), is it to know how long something will take (story points are a distraction), is it to see how much work a team can do in a sprint (remove the 1 day planning/retro and you get an extra day to deliver the work), is it to fit into a bigger team of dependencies (story points, again, are a distraction).
If you are using story points to avoid figuring out how long something will take, then be brave and figure out how long it will take.
(sorry turned a bit ranty!)