I've experienced scanning personal books and also try to reduce them since I'm also concerned with bloat on my (older) mobile reading devices. Unfortunately, there are reasons I cannot upload those, but the procedures might still be helpful for existing scans.

Use ScanTailor to clean them up. If there is no need for color/grayscale, have the output strictly black and white.

OCR them with Adobe Acrobat ClearScan (or something else, that is what I have).

Convert to black and white DJVU (Djvu-Spec).

Dealing with color is another thing, and takes some time. I find that using G'MIC's anisotropic smoothing can help with the ink-jet/half-tone patterns. But it's too time consuming to be used for books.