High-standards imposed by distros (see for example: https://www.debian.org/doc/debian-policy/) and tedious work of packagers has been a very important part of the free software ecosystem, and kept GNU/Linux snoopware-, malware- and virus-free and trustable [1]. They are far from "unneeded middleman".
Here's a very recent example of this system at work: Chromium sneaks a binary blob which secretly listens to your microphone and sends data to Google: https://lwn.net/Articles/648392/
This is just one instance, there has been many many examples of this.
There are just so many, let me say, shady, "apps" out there, so I don't install anything from Google Play Store on my smartphone. On my desktop/laptop however, I feel safe installing whatever I need from the official repositories. I personally don't want to use a distro to which "app" developers from Android ecosystem can freely push their programs (and updates to them).
> Applications being files/folders that can be installed/uninstalled with cp/rm seems way more unixy to me.
But that's essentially what a package manager does for you, although it takes away your freedom to screw up in the process. In GNU/Linux, it essentially 1) protects you from screwing up your system accidentally 2) allows you trust system-wide binaries (assuming that your distro is trustable).
App Stores are also package managers, so I'm not sure what you're trying to point out here.
====
[1] I'm excluding Ubuntu here. Ubuntu comes with spyware, spies on you by default and monetizes the privacy and trust of its users [2]. I don't trust it, I don't use it, and I recommend against it.