Very clever! Check out the Account Activity API [0]. You can have Twitter POST to a webhook whenever your tweets get a reply (among other things), letting you skip the polling approach to look for replies.
If it’s just one webhook, it’s within the free tier.