Your scenario makes me suspecting that it is Xilinx flavored Yocto causing the problem. I think that removing some unused Xilinx-specific layers/recipes can reduce the prologue and epilogue execution time.
I've tried asking the Xilinx community, and got only a reply saying that there is a database in Yocto which limits the scalability.