Responsive design isn't to blame at all. The "Javascript all the things" approach is to blame.
You can absolutely do a good responsive design in pure HTML+CSS. What seems to happen in practice, though, is pages that appear as flat white if I don't fiddle with my NoScript settings, and that grind on my (admittedly not top-of-the-line) computer if I do.