Creates a matrix with one axis in a given direction. Then a second axis at a straight angle to that, based on a reference up axis. The third axis is simply perpendicular to both. Position goes into the last vector of the matrix.
Edit: I think it has been answered by others - up is in world-space, and the camera is not necessarily aligned horizontally.
y = up.reject(direction).normalized()
a.reject(b) = a - b.project(a)