Taking a bus within the same city has never been a problem and these routes are typically pretty full. However, once you're going from one city to another, within LA county, the added time of transfers and making dozens of stops on city streets really starts to add up. A trip that might take 45 minutes by car sitting in traffic could easily take 2+ hours by bus. Nobody who values their time would choose the bus option if they can afford a car.
People don't avoid the bus because many of the passengers are downtrodden. Many of the passengers are downtrodden simply because anyone who isn't chooses the quicker more efficient option of driving.
The buildout of rail and the addition of more express buses I agree will help alleviate a lot of traffic issues. Buses can be the solution to the last mile issues. But like you said, there needs to be a lot more of them to cut down on waiting time.