https://linux.die.net/man/8/iptables
--pid-owner processid
Matches if the packet was created by a process with the given process id.
But was disappointed by the note:> NOTE: pid, sid and command matching are broken on SMP
That's a really outdated version of the man page. The current one is: https://ipset.netfilter.org/iptables-extensions.man.html
Probably the closest thing these days is to use the cgroup match extension, together with something that makes sure separate apps are moved to their own cgroup. But that cgroup would be under a separate parent cgroup for each user so it's not ideal.
I expect NFQUEUE or some EBPF magic is a better way to do things - you don't want to be constantly adding/removing per-process iptables rules.
The iptables rules can be converted directly to nft with iptables-translate and the syntax is more simple.
Firejail [2] and Bubblewrap [3] can put limits on application capabilities and/or take away network access. Firejail has default policies for many applications, all of which can be overridden by user configurations in their home directory. Firejail leverages Linux Capabilities and AppArmor. For example, when I launch VLC even if it were configured to pull down album information, it could not. If one day they added default-enabled telemetry to VLC it would go nowhere.
A round-about and heavy handed way to manage communications by PID would be to isolation applications in their own VM or container and map the VM or container to a unique user or group.
[1] - https://ipset.netfilter.org/iptables-extensions.man.html
But does it have any sense? Usually you block inbound connections, allowing only certain services. If a rogue process starts listening on a local port, you could display a warning alert, and as inbound connections are already blocked you'd be safe (as long as you trust netfilter...) and you wouldn't need to ask the user to perform an action.