That's a pretty poor justification. Their perspective is wrong, and their constraints don't prevent them from treating security bugs differently as they should.
On the level of the Linux kernel, this does seem convincing. There is no shared user space on Linux where you know how each component will react/recover in the face of unexpected kernel behaviour, and no SKUs targeting specific use cases in which e.g. a denial of service might be a worse issue than on desktop.
I guess CVEs provide some of this classification, but they seem to cause drama amongst kernel people.
There's been a lot of criticism written on the kernel devs stance over the last, what, 20 years? One obvious problem is that without giving security bugs, i.e. vulnerabilities priority, systems stay vulnerable until the bug gets patched at whatever place in the queue it happens to be at.