I was tempted to do something similar when I got my Yellow because the low availability of the CM4s back then, but it seemed that, at least with older Radxas there were problems with kernels or compatibility. It might work with the newer one if you say it's mainline kernel but haven't found anything in the forum in quick search...