we use that same model at our company, it powers not only our devs but also many business needs. We rent 1 gpu (B300), it costs around 10x less than the api costs
Do you rent from AWS or some other provider? I wonder if there are issues with on-demand rent for those high-end GPU instances, is capacity always there or sometimes it is unavailable?
I don't touch AWS. We rent from an european provider. Demand can be tricky for serverless/spot-pricing. We have an always-on server so, no issues.
If there's some cost or availability question you have, AWS has some complicated bullshit to solve the problem for you. Check out reserved instances.