I've used both substantially, Telegram for the past 2-3 years, Signal since it was Textsecure.
The primary concern is that messages in Telegram aren't encrypted by default. But that's been the case for a lot of messengers and tbh for large groups privacy really can't be assumed on any E2E solution. (yes, technically but practically it wouldn't be the case)
The Telegram creators are also extremely cocky and the encryption they do use is non-standard and done mostly in-house. It backfired on them a little with MTProto which they've fixed in v2, but it doesn't make cryptographers confident.
Signal and Telegram have wildly different philosophies on what it means to be secure. Telegram refuses to implement E2E on desktop clients citing it being too large of an attack surface (I am inclined to agree with them). They emphasize ephemerality of conversations in a way Signal doesn't do (E2E chats in Telegram and frequently brought up and torn down, Signal instead just has self-destructing messages).
Finally, look at the creators' motivations. Moxie is having to sell double-ratchet stuff to the likes of Facebook and relies on Amazon and Google to run the service. Pavel is several orders of magnitude more rich, has been outspoken against Putin, and can afford to fund anything he needs done on Telegram. I'm not claiming either is more trustworthy, but motivations are radically different.