Can you eat it?
(The FDA limit is 2% by mass of product, but it's silica. Silica is _extremely_ chemically stable.)
Also, and here's what I suspect the real reason is; silica gel sold as a dessicant is hard (not quite as hard as silica glass, but hard), in relatively large particles, and indigestible. That's mechanically bad news for your intestines.