The vast majority of this traffic is from Manhattan - otherwise it wouldn't be spread across so many routes. If you want to designate a cars-ok route in Manhattan between the Battery Tunnel (to Brooklyn) and the Holland Tunnel and let that back up, you could do that and it would be less of a mess than the status quo: it would stay on West Side Highway out of the way and it would have less traffic.
One time I took a taxi from Newark to my home in Brooklyn, and the taxi driver dropped me off right after getting out of the Holland Tunnel (next to a subway stop) saying he didn't want to deal with more traffic. There is no effective vehicular link between New Jersey and Brooklyn right now. There is a moderately effective link between each of those and Manhattan, that's it.