Browsers shouldn't set a preferred colour scheme by default.
I think the prefers-color-scheme media query would be respected on more sites if by default it had the value "unset" or something, instead of defaulting to "light" or "dark".
I personally don't respect it on my sites for this reason. 99% of people visiting my sites won't actually have set this value themselves.