Readability (https://github.com/luin/readability) is a classic, and included as part of Firefox (I think, maybe that's been discontinued). It's essentially a bag of hand-written heuristics but they're pretty good heuristics.

Some interesting reading is Christian Kohlschütter's thesis on this problem, which is framed in academia as "how do we assemble good text corpuses from webpages for data analysis, which means removing junk (boilerplate) from our HTML crawls" (https://code.google.com/archive/p/boilerpipe/wikis/WSDM2010P...). Boilerpipe would probably be the right way to go, but if you're not using Java it could be harder to integrate.