The 3DS, based on hardware/system alone, should have failed. It was tremendously underpowered, and it forced a terrible 3D technology on all its users, which never really took off or became anything other than a novelty while raising the cost of manufacturing.
The Vita was a truly next-generation portable gaming device, with a beautiful screen, incredible graphics, properly analog sticks, an extremely modern interface, and great connectivity and human-interface (camera, capacitative touch) features. Even incorporating the cost of a proprietary Sony memory card, for the amount of power you got, I think it was very reasonably priced. Comparing the 3DS and Vita was like night and day in terms of a modern gaming device. Even today, I think it can stand head-to-head with the Switch in terms of portable gaming.
But Nintendo continued to pour resources into developing top-tier games for the 3DS, slashing its price to bolster adoption, and sticking to its still unorthodox 3D screen/touch screen combo. Whereas for the Vita, Sony quickly got spooked that the Vita didn't perform as well as the PSP, pulled first-party support and general marketing support, and major 3rd party developers (particularly outside Japan) fled the device.
The 3DS is now seen as a major success for Nintendo, while the Vita died (or is still dying) a slow and unceremonious death.
Sometimes sticking to your crazy guns works.