Yep, the up vector is not necessarily orthogonal to the direction vector (which is also called “look at” vector in OpenGL [1]). Another approach would be to set

    y = up.reject(direction).normalized()

where

    a.reject(b) = a - b.project(a)

[1] https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml...

We've already defined z and x by that stage, and we need y to be a unit vector perpendicular to both. So y can only be z.cross(x) (or x.cross(z) for mirrored).