Also there is aspect of defence in depth. Maybe you can compromise one package that itself can't do much, but installer runs with higher priviledges and has network access.
Another angle -- installed package may compromise one container, while a bug in apt can compromise the environment which provisions containers.
And then at some point there is "oh..." moment when the holes in different layers align nicely to make four "bad but not exploitable" bugs into a zero day shitshow