If you get an ECAD/EDA tool you could do some work to reduce the size of the board.

From what it looks like, you can squish things together by a lot still and replacing components could further reduce it.

Though it's unlikely you'll be able to match the Yubi Nano easily unless you go for some expensive decisions (expensive unless you get a couple thousand keys).