If you were really a "freelancer" or "consultant" instead of a contract developer, you would have these relationships yourself. You would be able to convince your clients to cut out the middleman and pay you $100/hr instead of paying me $120/hr. But first, you have to build that trust with the people ultimately paying for your services. This generally means upping your game with respect to salesmanship, which isn't something you can "hack". It also means turning down opportunities that may pay well, but don't offer any opportunity for building relationships.
Again, that's the thread that's missing here: personal relationships. Get to know your clients, understand what needs they have, and make them like you enough as a person that they think "Hey, I need a developer. I wonder if ryannevius is available?" You and your friend had a hard time starting an agency because you don't have a pool of potential clients because you don't have that symbolic rolodex of people you can call. You're selling professional services, and professional services are sold through person-to-person relationships.