Yes, security updates absolutely matter regardless of your usage pattern. You could unwittingly visit a website that has been compromised to exploit a zero-day in your browser. Or receive a malicious text message. Or open a PDF. Or or or.
I am not defending unpatched phones just to be clear, but its not end of the world if you use unsecure device, just keep all your money and other important stuff away from it. Which is fine for many people.
In Android, browser, messaging app updates and many even system updates are delivered through Play store (long after system/OS updates have stopped for the phone), so attacks will have to be much more sophisticated.