You've been pretty vague so far, so it's hard to offer concrete advice. A few thoughts, though:

>I hear the words "why not both" in my head, but I'm not sure how to pull that off.

In my time working with open source, I have found only one good way to monetize an open source project: hosting. Keep it simple: users can pay $x/month to access your service on your servers, or they can host it themselves for free. If you really want to turn up the heat, offer direct support tickets and feature requests for your paid users, but do not pull punches on your self-hosted version. Ideally, you can have your cake and eat it too: software like Grafana and Obsidian.md have gotten away with this for a while, and they've seen the benefits of both open-source and business customers. It's a beautiful thing, if you can pull it off.