Quarto can do this really well (eg, my blog here: https://andreasthinks.me/posts/burglary_attendance/). It also does Rmarkdown.

The built-in nbconvert tool can already do that. Maybe not the most beautiful output by default, but you can give it custom CSS or latex template (for PDF output) via CLI flags.