Cool project! Last year I built self driving robot for my bachelor's thesis. Instead of building end-to-end deep learning pipeline I used two neural nets: one trained with genetic algorithm to drive a robot based on ultrasonic sensors, another for object recognition and detection. Based on detected items (like road signs) robot took different actions.
Video: https://www.youtube.com/watch?v=cUXh7iP3hoQ Code: https://github.com/kazepilot