What browser you use? I suppose it is Chrome/Chromuim-based browser of Firefox.
Did you read its changelogs? There are a lot of JavaScript-related vulnerability issues fixed this year (as in previous 10 years), right?
There are of course exceptions, those in particularly sensitive situations, but in those cases other precautions should be taken as well.
And yes, the text based browser you use may see fewer security patches in the change log, but it is also far less tested. The user base is just not there. If no JavaScript is that important to you it can easily be disabled in Firefox and I'd wager you'd come out ahead in terms of security.
JFTR, Is not Firefox GUI itself uses/based on JS?[0]