How do you train it for emergency situations (i.e. a car suddenly turning left in front of you)? I'd imagine it would be hard to get many of those in the training data set.
And if driving in highway is a problem, why not use a test terrain complete with fog generator? With RC car representing pedestrian, other car, animal, ... feed the video first into an AR system and then give it to the neural net.
anyway good hack, I wish you well :)