Most people in financial jobs also harm no one, and though they accept higher wages, they also create far more value, and there could be endless arguments over whether or not there is trickle-down value, whether managing pension plans adds value, etc.
But what concerns me more is that most employees in any organization exist solely for the short term gain of their bosses (somewhere up the food chain). A fourth-grade teacher exists more to make a principal look good for advancement to superintendent than for endowing society with the value of educated children. The fourth grade teacher has to put up with arbitrary state standards, political fighting between staff, inappropriate parental involvement, turning the other way on some issues they feel strongly they should not turn the other way on. It's a political shitstorm like anything else, where a few people on top of something (no matter how small scale that something may seem by comparison to a huge bank) are applying pressure to make the teacher's job more about subservience and fealty than about efficacious results.
I think that problems get worse and worse as you go up the academic and research food chain. You might be able to make an argument that being a pre-school teacher in a small rural town, or the pastor of a small community's church (modulo one's beliefs about whether promoting religious faith can ever be morally or ethically good), are actually "ethical" jobs, but I'm not totally sold that you can, and I don't believe much beyond this can even enter the discussion.
It's not so much about whether you overtly participate in something wrong. It's about whether the aggregation of all the tiny little microscopic job duties you do to comply with bureaucratic hierarchy ends up summing up to unfair gains mediated towards someone who doesn't economically deserve them, or that the gains end up being wasted.