As for non-fiction popular press books, I've been told that a typical advance can range from less than 10k to around 50k-60k - the idea being that this money will sustain the author for the 6 months to a year or so it takes to finish the manuscript. (Would that this were feasible for academic books!)
The real reason college textbooks are out of control has nothing to do with professors getting rich, they aren't. They're out of control because publishers are doing professor's grading in exchange for the professor just assigning the book.
Instead of a publisher selling a book for e.g. $70, they sell it for $100 and bundle with it a "one time use code." This code allows students to log into a publisher run website and turn in assignments, complete problems, and so on.
So effectively the professor has outsourced a large chunk of grading to a third party and better still (for them) it is "free." And all they have to do is assign the $100 book instead of the $70 one, then just log into a free portal and collect all of the grades for the semester.
Publishers naturally get very rich off of this. They only have to set up a web-site once and then they can resell it to dozens of different schools, hundreds of classes. Plus they really cut corners with the grading and questions too (often out-sourcing those to the original authors of the book as part of the contract to update the book annually).
The only way to solve this is to put pressure on colleges, departments, and professors. Publishers are doing what they're meant to: trying to make every cent they can. It is professors and colleges which are failing in their duty and effectively outsourcing part of student's education to a third party for a fee (which the student pays).
The only other thing I can see resolving it is states stepping in and essentially making rules which say that the tuition fees must be all inclusive (e.g. you must get given the textbooks on the first day of class, but the cost of those is included into the main fee so there is nothing hidden).
So even if this scheme continued it would all be out in the open, and students could accurately compare the fee structure between two schools, one that is taking part, and one which is not...