For a drone (even a swarm of them) to compete with a modern jet fighter, it'll need a reasonably powerful engine and sufficient fuel to keep it up in the air long enough to: seek out and engage the enemy, carry a decent payload, and reach a suitable altitude. That all sounds like a lot of fuel and engine power to me, especially if you're expecting this drone (swarm) to operate outside of the immediate vicinity of the launching area.
I see more potential for drones to act as 'screen' for jet fighters once stealth technology is made irrelevant by advances in radar. I foresee the use of, low cost vehicles that resemble fighters (in the ways that count), but carry minimal ordinance (if any) to keep the costs down. They could also provide auxiliary functionality like electronic warfare and scouting.
I mean you could in theory develop a drone that exploded on impact and operated within a limited area, but you've just re-invented the guided missile, with a little extra smarts.