There may have to be some clever system where you don't take a lot of fuel with you.
Imagine a type of particle accumulator and aggregator - call it cosmic flypaper. If you're going say 100 km/s, things seemingly insignificant and rare you may be able to collect a lot of pretty quickly.