It's not a replacement for TeX. R originally had Sweave for weaving R and text together. Then we got Rmarkdown, which introduced the use of markdown for authoring documents via pandoc, but the output document only had simple features like headers, sub headers, emphasis and so on.

Now Yuhui from Rstudio (who's been prolific in this area) has written bookdown for authoring books with rmarkdown. It's not for authoring statistical papers (as the other comments stated - that's rmarkdown's realm) it's for books like Hadley Wickham's R4DS and Advanced R. Other examples can be found on the bookdown website.

I'd also add that, regardless of inserting R code, bookdown is a class authoring format and personally I would write a book or large document in bookdown even if it was just pure prose.