Both lost significant deposits from crypto blow-ups, reducing their liquidity and tanking their stock prices [1]. That left them uniquely vulnerable to both perceptions [2] and rates (as they sold the short-dated, liquid, low-duration assets first to plug the crypto hole).
[1] https://www.bloomberg.com/news/newsletters/2023-03-09/silver...
[2] https://www.barrons.com/articles/crypto-silvergate-signature...