After moving here 2 years ago, I had to study for the driver's license written test and I remember being shocked at reading "when turning at an intersection, if you are in the only turning lane, you may enter into any of the lanes during your turn".
So I'm imagining that if you both had green lights, since you were turning left, he had the right of way (not saying his actions weren't stupid by the way).