It's definitely better to have a strong LOR from a PI, even moreso from a PI at the school to which you're applying. "No great LORs but self-publishing" can be shortened to "no great LORs."

Why do you see these things as mutually exclusive? You could self-publish and use it as an "in" to working for a PI over the summer. If you do well and the PI likes you, they likely have a lot of sway with admissions and can help you get into whatever grad program they work with.

Disclaimer: Never applied/went to grad school but know many people who did... I've seen the strategy I described work at a high tier school.