1. Search for x (or y).

2. Ask for directions

3. Replace your location with y (or x).

4. Reverse start and end if necessary.

Personally I think is flow is better than directly entering start and end for two reasons:

- The common case (go from where I am to somewhere else) is faster.

- I don't have to think about how to enter "my location", which isn't obvious (over the years I've seen - typing "my location", searching with the start box empty, pressing a special button for the purpose, and others).