No freedom is perfectly available. Freedom of speech, for example, doesn't allow you to commit perjury or fraud, and it doesn't entitle you to harass someone. Freedom of association doesn't let you stalk somebody, or let felons on parole talk to one another. All rights must be balanced against other rights.
Businesses do have a very broad right of freedom to serve who they please. The one exception is when historically many businesses have used that power to participate in oppression against specific downtrodden groups. So a business can happily decline to serve US congressmen, freemasons, and people who have thrown up on their floor previously. But they can't decline to serve black people as a group.
The specific groups who have been legally recognized for this are called protected classes, and it's pretty settled law in the US: https://en.wikipedia.org/wiki/Protected_group