What if a public function is very complicated—too complicated to be considered a unit—and it calls several simpler private functions? Those helpers should be tested with their own unit tests, but shouldn't be exported.
It's not just about coverage in terms of code execution; you also want the right test “resolution”, so you can quickly find the level of abstraction where a bug is, and so you can prove correctness of the parts of a complicated function as you build it.