I normally browse with a browser that doesn't even support JS (Lynx, NetSurf, or Dillo). Glad to know that anyone who doesn't use a browser developed by a company running on billions of dollars a year is just being "silly".
If I am compelled to use a "modern" browser, I turn off JS, cookies, remote fonts, WebGL, and any third-party resources and enable them on a case-by-case basis if the website is important enough (it usually isn't). Most sites worth visiting work much better when I do this.
I have Javascript off by default and only turn it on (using NoScript extension) whenever it's actually needed.
Most pages load just fine without and with those that don't it's a 50/50 between me enabling JS for it or deciding I didn't wanna view the content anyways and closing the tab.
As someone else mentioned it's not to deprive myself of functionality, but to deprive the vultures (trackers and other shady stuff) of it.
Not that I personally disable all JS in my browser, but I'd say if your website can not be displayed without Javascript, then it is silly.
Javascript's terrible when it's used to generate pages from bloated frameworks that create 5000 DOM elements, add listeners to everything, load a dozen external scripts, and so on, but it's really valuable when used to actually improve the user experience.
Rather, they do so because it enables a majority of websites, and this includes big names like news websites, to create an absolutely horrible user experience - even if, or seemingly because, the content profits in no way from JS.
If you have ever tried to surf on an older laptop recently, you will get what I mean.
Only three engines exist that work well with JS; if we want our sites to not be dependent on behemoths like Google and Apple (Google is Mozilla's income source), we need to build sites that don't depend on their software (Blink, Webkit, and Gecko). That typically means not requiring JS.
People who block JS do so for good reason: when they open your site, they have no reason to believe that the JS being served isn't malware. If they disable and inspect it, they have no reason to believe that the scripts won't change the next time they open your page. It's safer to just leave it disabled.